Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 22.864952127s FAIL (2025-07-02 04:40:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 22.864978669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.76507103s FAIL (2025-07-02 04:39:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.765097911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.640113098s FAIL (2025-07-02 04:38:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.640146272s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.980078234s FAIL (2025-07-02 04:37:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 9.98012307s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.616322926s FAIL (2025-07-02 04:36:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.616354495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.592419675s FAIL (2025-07-02 04:35:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.592450474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.738611809s FAIL (2025-07-02 04:34:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.738640935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.552849317s FAIL (2025-07-02 04:33:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.552875036s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.675614123s FAIL (2025-07-02 04:33:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.675636548s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.723911606s FAIL (2025-07-02 04:32:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.723935874s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.660229553s FAIL (2025-07-02 04:32:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.660255882s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.739731829s FAIL (2025-07-02 04:31:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.739761867s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.670374521s FAIL (2025-07-02 04:30:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.670405021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.656173716s FAIL (2025-07-02 04:29:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.65619765s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.602081316s FAIL (2025-07-02 04:28:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.60210369s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.683773222s FAIL (2025-07-02 04:28:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.683813819s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.657178137s FAIL (2025-07-02 04:27:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.657199367s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.743230743s FAIL (2025-07-02 04:27:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.74325485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.60939713s FAIL (2025-07-02 04:26:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.609424042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.676695859s FAIL (2025-07-02 04:25:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.676766585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.686855562s FAIL (2025-07-02 04:24:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.68688077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.666971033s FAIL (2025-07-02 04:23:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.666994949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.790523546s FAIL (2025-07-02 04:22:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.790552652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.652347999s FAIL (2025-07-02 04:22:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.652371203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.176416653s FAIL (2025-07-02 04:21:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.176445668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.658462633s FAIL (2025-07-02 04:20:58 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.658487541s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.783008648s FAIL (2025-07-02 04:20:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.783035047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.596405824s FAIL (2025-07-02 04:19:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.596427186s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.713730532s FAIL (2025-07-02 04:18:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.713757424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.697516914s FAIL (2025-07-02 04:18:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.697543344s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.683817602s FAIL (2025-07-02 04:17:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.683847668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.658902129s FAIL (2025-07-02 04:16:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.658930193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.767884511s FAIL (2025-07-02 04:15:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.767909458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.784026983s FAIL (2025-07-02 04:15:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.7840519s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.096407186s FAIL (2025-07-02 04:14:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.09643531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.637921677s FAIL (2025-07-02 04:14:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.637946103s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.237101924s FAIL (2025-07-02 04:14:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== ush batch item: pb.BatchRequestItem_SegmentCommit Jul 2 04:12:26.387385353 (211h2m1.158026679s) [upload=168817]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 2 04:12:26.675658004 (211h2m1.446299329s) [upload=168817]: Done uploading segments... Jul 2 04:12:26.675673784 (211h2m1.446315119s) [upload=168817]: Upload finished. err: <nil> Jul 2 04:13:31.474556357 (211h3m6.245197692s) [upload=168832]: Starting upload Jul 2 04:13:31.474610561 (211h3m6.245251896s) [upload=168832]: Done starting upload Jul 2 04:13:31.474732024 (211h3m6.245373359s) [upload=168832]: Uploading segments... Jul 2 04:13:31.474790095 (211h3m6.24543143s) [upload=168832]: Got next segment. Inline: false Jul 2 04:13:31.474814822 (211h3m6.245456157s) [upload=168832]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 2 04:13:31.474819090 (211h3m6.245460415s) [upload=168832]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 2 04:13:31.711289277 (211h3m6.481930612s) [seg_pos={0 0}, upload=168832]: Begin upload segment... Jul 2 04:13:31.711327530 (211h3m6.481968865s) [seg_pos={0 0}, upload=168832]: Done begin upload segment. Jul 2 04:13:31.711344092 (211h3m6.481985437s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711350844 (211h3m6.481992179s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711381894 (211h3m6.482023229s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711712867 (211h3m6.482354192s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711721924 (211h3m6.482363259s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.711965660 (211h3m6.482606995s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712207002 (211h3m6.482848337s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.712474514 (211h3m6.483115839s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.713982268 (211h3m6.484623603s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714206547 (211h3m6.484847882s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714452207 (211h3m6.485093542s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714704500 (211h3m6.485345825s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.714963496 (211h3m6.485604831s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715250605 (211h3m6.48589194s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715546020 (211h3m6.486187356s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:31.715784957 (211h3m6.486426282s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Uploading piece... Jul 2 04:13:33.312716396 (211h3m8.083357731s) [upload=168832]: Next returned nil segment Jul 2 04:13:33.312734039 (211h3m8.083375374s) [upload=168832]: Waiting for error group managing segments... Jul 2 04:13:33.599257192 (211h3m8.369898517s) [noise=true , address=104.36.82.11:4115 , piece_id=ZUETVJUM635YFQ5O2DL2XOOV2356MIPCRCRWZEYTGP2CIPBVC7GQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.640829223 (211h3m8.411470559s) [noise=true , address=104.36.82.9:4006 , piece_id=NEJ2UKW4EYJWCIQZNPF5GBM3K46NV4HSELYXJCHYHJXOUMU3LNAA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.641404835 (211h3m8.41204617s) [noise=true , address=104.36.82.10:4079 , piece_id=DMRPZ3TANJSQT64YS2KA5CL2QBZ4RH5XR3UCLAWT32EDIAXGJYQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.750428801 (211h3m8.521070136s) [noise=true , address=104.36.82.8:4972 , piece_id=HWT7SJ2WHFPFJS46HN4TLFTOGJYD6PDR73NVUJTCQVOHJLSUZGOA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.759426104 (211h3m8.530067429s) [noise=true , address=104.36.82.5:4837 , piece_id=5MM32E4WUSDJWFJMWYH2DVAE3LHH35CGESV7BKCO6QVELZGJXNZA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:33.813623400 (211h3m8.584264735s) [noise=true , address=104.36.82.7:4912 , piece_id=434QVIDL2ZWSJQAWLXC5AB2EJI4QJUMUS3H5WAQLMQWV5NJFDQZQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:52.644497091 (211h3m27.415138426s) [noise=true , address=77.90.31.18:4805 , piece_id=GSFSJM6H6PBJBWELLNCT5RRYM6GLX5DVUMTKGNN5GUG4SN7IBJNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:53.416464828 (211h3m28.187106163s) [noise=true , address=45.91.116.11:4118 , piece_id=S5CIRWRTJGL25WZ3J6GT5GRDVTGDXQ6JOJLEYR2FSENY3TFBJH5A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:54.268176565 (211h3m29.0388179s) [noise=true , address=45.91.116.9:4261 , piece_id=S6NZB6X2P5VDCA3PFUOWJNK2YNGGOWCPZYD7WK3TRKFPBOMR7C2Q, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.205115451 (211h3m29.975756786s) [noise=true , address=45.91.116.10:4082 , piece_id=MPFJWXUJ3NDDLTWB2VVQPKWZAFNC4FZQDE6SHJJPW7KXKCAW37FA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:13:55.235263728 (211h3m30.005905063s) [noise=true , address=45.91.116.5:4298 , piece_id=LLYFJ6GPPMODLRE5LZZJXQOVQKOVJ6EVKKIKPMCMRVND7H2WBEQA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: <nil> Jul 2 04:14:00.823234032 (211h3m35.593875367s) [noise=true , address=45.91.116.5:4297 , piece_id=ED6SPYTR22NV6LGT73IHJJLSQ25EAQJMDPXX2GZSVW7H2GFQRU6A, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, address:45.91.116.5:4297): protocol: expected piece hash; context deadline exceeded; EOF Jul 2 04:14:00.823299347 (211h3m35.593940682s) [noise=true , address=77.90.31.16:4734 , piece_id=QX6ZBTJUA6KH5PF6BWHTIRGJ3R46RE4A4W6TCMRO76DEBVWZTNXQ, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: write tcp 188.166.225.170:32986->77.90.31.16:4734: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823310819 (211h3m35.593952154s) [noise=true , address=77.90.31.14:4661 , piece_id=NDED32M2DJKMKSLH6MWQHMTS6QWTFGPPXD4HUNBQ4AZY2XAJHNBA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: write tcp 188.166.225.170:51240->77.90.31.14:4661: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 2 04:14:00.823384240 (211h3m35.594025575s) [noise=true , address=45.91.116.12:4226 , piece_id=FORMLEQITRYKTXG6ZA2DAKPZXBE2GKEDPRG43G7EUTZJGANRMGIA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56388->45.91.116.12:4226: use of closed network connection Jul 2 04:14:00.823396292 (211h3m35.594037627s) [noise=true , address=77.90.31.17:4772 , piece_id=EEQNJFWGKQ6MDIWMFYYN4WXJBT4DIIK6W7242P5RH7YNNEXBZTNA, seg_pos={0 0}, upload=168832]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58032->77.90.31.17:4772: use of closed network connection Jul 2 04:14:00.823414087 (211h3m35.594055422s) [seg_pos={0 0}, upload=168832]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 2 04:14:01.058258351 (211h3m35.828899686s) [upload=168832]: Done uploading segments... Jul 2 04:14:01.058282947 (211h3m35.828924282s) [upload=168832]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.237124957s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 174.88833ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:14:16 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 174.919598ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 2 out of 60 is failed (threshold=6) [XX..........................................................] duration: 260.495585ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:13:40 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 260.51388ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 3 out of 60 is failed (threshold=6) [XXX.........................................................] duration: 179.571038ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:12:56 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 179.606004ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 4 out of 60 is failed (threshold=6) [XXXX........................................................] duration: 179.867576ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:12:20 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 179.902602ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [XXXXX.......................................................] duration: 173.0082ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:11:55 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 173.038298ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [.XXXXX......................................................] duration: 176.64997ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:11:17 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 176.698833ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [..XXXXX.....................................................] duration: 177.484796ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:10:12 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 177.513011ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [...XXXXX....................................................] duration: 182.208151ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:09:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 182.230634ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [....XXXXX...................................................] duration: 173.061724ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:08:39 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 173.099526ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [.....XXXXX..................................................] duration: 172.253297ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:07:38 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 172.283886ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [......XXXXX.................................................] duration: 174.539943ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:06:54 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 174.558608ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [.......XXXXX................................................] duration: 180.456341ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:06:03 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 180.479165ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 5 out of 60 is failed (threshold=6) [........XXXXX...............................................] duration: 178.351907ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt FAIL (2025-07-02 01:05:37 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 178.376965ms edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)