Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X...X.....................................] duration: 6.703713503s FAIL (2025-09-27 13:09:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.703746415s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X...X....................................] duration: 12.661621991s FAIL (2025-09-27 13:08:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 12.661671605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 11.047424203s FAIL (2025-09-27 13:08:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 11.047451955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 7.819383766s FAIL (2025-09-27 13:08:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.819554271s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.249229804s FAIL (2025-09-27 13:07:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.249267364s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 6.438756189s FAIL (2025-09-27 13:06:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.438790654s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 8.645372103s FAIL (2025-09-27 13:06:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.645418652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 9.111313882s FAIL (2025-09-27 13:05:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 9.111345804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 8.818842078s FAIL (2025-09-27 13:04:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.818877235s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 6.792918867s FAIL (2025-09-27 13:04:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.792957331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 11.114266643s FAIL (2025-09-27 13:03:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 11.114296229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 6.637418203s FAIL (2025-09-27 13:02:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.637445436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 8.558672833s FAIL (2025-09-27 13:01:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.558700436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 8.98081711s FAIL (2025-09-27 13:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.980850242s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 7.017847566s FAIL (2025-09-27 13:00:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.017883866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 7.479322848s FAIL (2025-09-27 12:59:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.479354509s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 9.342186219s FAIL (2025-09-27 12:59:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 9.342222167s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 7.499184365s FAIL (2025-09-27 12:58:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.499219111s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 7.09984521s FAIL (2025-09-27 12:57:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.09987193s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X...X..................] duration: 6.876327098s FAIL (2025-09-27 12:56:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.876355171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................X...X.................] duration: 11.093354318s FAIL (2025-09-27 12:55:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 11.093384916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X...X................] duration: 6.069692177s FAIL (2025-09-27 12:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.069722253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................X...X...............] duration: 6.377214119s FAIL (2025-09-27 12:54:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.377248504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 10.006548711s FAIL (2025-09-27 12:53:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 10.006581132s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 6.427204183s FAIL (2025-09-27 12:52:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.427233489s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X...X............] duration: 8.232282256s FAIL (2025-09-27 12:52:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.232322463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................................X...X...........] duration: 6.716482114s FAIL (2025-09-27 12:50:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.716514707s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X...X..........] duration: 8.335718554s FAIL (2025-09-27 12:49:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.335747668s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X...X.........] duration: 7.525789125s FAIL (2025-09-27 12:48:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.525818933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X...X........] duration: 6.256999655s FAIL (2025-09-27 12:47:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.257031004s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X...X.......] duration: 7.755371511s FAIL (2025-09-27 12:47:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.755406139s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X...X......] duration: 8.329404461s FAIL (2025-09-27 12:46:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.329436781s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................................................X...X.....] duration: 7.198195528s FAIL (2025-09-27 12:44:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 7.198226616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.................................................X...X....] duration: 6.576119279s FAIL (2025-09-27 12:44:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.576152293s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.................................................X...X...] duration: 15.698789066s FAIL (2025-09-27 12:43:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 15.698817672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.................................................X...X..] duration: 6.371548717s FAIL (2025-09-27 12:42:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.371582411s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.................................................X...X.] duration: 6.523478204s FAIL (2025-09-27 12:41:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.523587984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.................................................X...X] duration: 30.244061841s FAIL (2025-09-27 12:41:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:40:38.562009574 (95h9m24.024102047s) [upload=83717]: Starting upload Sep 27 12:40:38.562097762 (95h9m24.024190245s) [upload=83717]: Done starting upload Sep 27 12:40:38.562155331 (95h9m24.024247814s) [upload=83717]: Uploading segments... Sep 27 12:40:38.562413553 (95h9m24.024506036s) [upload=83717]: Got next segment. Inline: false Sep 27 12:40:38.562456485 (95h9m24.024548968s) [upload=83717]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:40:38.562462256 (95h9m24.024554739s) [upload=83717]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:40:38.809537815 (95h9m24.271630328s) [seg_pos={0 0}, upload=83717]: Begin upload segment... Sep 27 12:40:38.809611835 (95h9m24.271704308s) [seg_pos={0 0}, upload=83717]: Done begin upload segment. Sep 27 12:40:38.809640250 (95h9m24.271732723s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.809651381 (95h9m24.271743864s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810155852 (95h9m24.272248325s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.810629083 (95h9m24.272721566s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.811192586 (95h9m24.273285069s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.812766944 (95h9m24.274859427s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.813825871 (95h9m24.275918344s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.814428858 (95h9m24.276521342s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.816165556 (95h9m24.278258029s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.817752558 (95h9m24.279845041s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.820049742 (95h9m24.282142225s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.822175891 (95h9m24.284268364s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.823864976 (95h9m24.285957459s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.824678837 (95h9m24.28677132s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.825683920 (95h9m24.287776403s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:38.829240292 (95h9m24.291332776s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Uploading piece... Sep 27 12:40:40.517955198 (95h9m25.980047671s) [upload=83717]: Next returned nil segment Sep 27 12:40:40.517975026 (95h9m25.980067509s) [upload=83717]: Waiting for error group managing segments... Sep 27 12:40:40.793448910 (95h9m26.255541383s) [noise=true , address=145.79.227.3:4150 , piece_id=TYXUYRUJ5I763P6LA3ZWWJ57BAK5ERBFHGWDCCO5INCNO3TSBJLA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.809315233 (95h9m26.271407706s) [noise=true , address=145.79.227.4:4186 , piece_id=6ECNLYOYFTVMFVEOI7JPGC4N6GC73TDVMINMIBSFSX3XDJHYWJCQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.819983888 (95h9m26.282076371s) [noise=true , address=145.79.227.11:4730 , piece_id=RQ4W27NDDXYNDASF3RUFS3EHKZQGNVREPKPFCPWZO7CUXPFE32ZA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.825146579 (95h9m26.287239052s) [noise=true , address=145.79.227.19:4077 , piece_id=PZES3PIFUG5BPASSA4AB7SKQHHFFI2F2R7IUAO6IEFSOKXIYJP5Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.841531729 (95h9m26.303624212s) [noise=true , address=145.79.227.7:4293 , piece_id=JJMI533NYTNLIZ3TB5F7YTVJWSXZTVXK53ATIOP2IFGRRPWHGGNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.854031301 (95h9m26.316123774s) [noise=true , address=145.79.227.16:4909 , piece_id=L25D4RPIOKOTA5STHH7AVBV4SFNPNL4FBL4ZUR7CQJCJMJDRVS4Q, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:40.855780121 (95h9m26.317872594s) [noise=true , address=145.79.227.5:4222 , piece_id=LWZSM7MXQ5OULE6VH4E4Z7L6M4J67O3EJ4W7YMYCNPJFA7CWHQBA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.094322992 (95h9m26.556415475s) [noise=true , address=77.90.31.19:4841 , piece_id=V7QRKA6UHRPNCFWZRA2FZJLGRKJGCVFFONZZDHZQRE4H7CTZCLDA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.137933835 (95h9m26.600026318s) [noise=true , address=77.90.31.12:4590 , piece_id=7KAH42T4BT67MIRCV2G6L2CH3IHGP4NBPYD64R7VYI7552BEAAZQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:40:41.429391216 (95h9m26.891483699s) [noise=true , address=77.90.31.11:4555 , piece_id=37KARAEKEA7OCRTJJIVGZLO2KB2I4UT6FZIBW2OXRUO2OYL5JMAQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: <nil> Sep 27 12:41:07.862946034 (95h9m53.325038517s) [noise=true , address=45.91.116.10:4082 , piece_id=HTBKFN2DNSDBR66EZKQU6N3XUQNFNE5EGVVN63LGA7XVWSE5FFDQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863078276 (95h9m53.325170759s) [noise=true , address=45.91.116.8:4048 , piece_id=TEWFOS4FVPPTHPMSCJWTQ2D5WE6TCLUXORGIHSN2HNS6U5RHODLQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, address:45.91.116.8:4048): protocol: write tcp 188.166.225.170:55962->45.91.116.8:4048: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863119715 (95h9m53.325212208s) [noise=true , address=45.91.116.5:4298 , piece_id=XHDFZIXFNEYI35IL2Z2MQ7A537ZOZ3WH4AN7L6ULOF6VEUSDHESQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C, address:45.91.116.5:4298): protocol: rpc: dial tcp 45.91.116.5:4298: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863214756 (95h9m53.325307239s) [noise=true , address=45.91.116.9:4261 , piece_id=2YAWMTOUBPONA3YYNY53LMZRPDDPDBTXSN2YCB6HDXW6A4NC6V3A, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:48564->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863322230 (95h9m53.325414714s) [noise=true , address=45.91.116.6:4973 , piece_id=SQOM4OVCG7OFOF3MQJNJA7JTRJW2IO6Y4HRM65P5KLXV64F2KBNQ, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:53336->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863430527 (95h9m53.32552301s) [noise=true , address=45.91.116.4:4939 , piece_id=HOVTKLVN5SQX6VFH5QZEXQZRSV6NV2D2YE5CL64HYZ6MCOB6CDHA, seg_pos={0 0}, upload=83717]: Done uploading piece. err: ecclient: upload failed (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX, address:45.91.116.4:4939): protocol: write tcp 188.166.225.170:44714->45.91.116.4:4939: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:41:07.863466505 (95h9m53.325558988s) [seg_pos={0 0}, upload=83717]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:41:08.104416400 (95h9m53.566508883s) [upload=83717]: Done uploading segments... Sep 27 12:41:08.104448861 (95h9m53.566541355s) [upload=83717]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.244090676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.................................................X...] duration: 16.376367948s FAIL (2025-09-27 12:39:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 12:37:13.690416760 (95h5m59.152509243s) [upload=83668]: Starting upload Sep 27 12:37:13.690515829 (95h5m59.152608312s) [upload=83668]: Done starting upload Sep 27 12:37:13.690666315 (95h5m59.152758808s) [upload=83668]: Uploading segments... Sep 27 12:37:13.690817974 (95h5m59.152910457s) [upload=83668]: Got next segment. Inline: false Sep 27 12:37:13.690869171 (95h5m59.152961654s) [upload=83668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:37:13.690876305 (95h5m59.152968788s) [upload=83668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:37:13.937266910 (95h5m59.399359393s) [seg_pos={0 0}, upload=83668]: Begin upload segment... Sep 27 12:37:13.937340380 (95h5m59.399432863s) [seg_pos={0 0}, upload=83668]: Done begin upload segment. Sep 27 12:37:13.937372270 (95h5m59.399464753s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.937451391 (95h5m59.399543884s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938081951 (95h5m59.400174444s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938563849 (95h5m59.400656332s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938965585 (95h5m59.401058068s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939353333 (95h5m59.401445816s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939739498 (95h5m59.401831981s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940144499 (95h5m59.402236982s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940668768 (95h5m59.402761261s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.942468184 (95h5m59.404560667s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.944294502 (95h5m59.406386985s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.945989398 (95h5m59.408081891s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.947618120 (95h5m59.409710603s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.949213799 (95h5m59.411306282s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.950845245 (95h5m59.412937728s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.952449149 (95h5m59.414541632s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:15.529577113 (95h6m0.991669586s) [upload=83668]: Next returned nil segment Sep 27 12:37:15.529600238 (95h6m0.991692711s) [upload=83668]: Waiting for error group managing segments... Sep 27 12:37:15.909365373 (95h6m1.371457846s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932137335 (95h6m1.394229818s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932507209 (95h6m1.394599692s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.938051066 (95h6m1.400143539s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.941172539 (95h6m1.403265022s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.944182742 (95h6m1.406275225s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.107915721 (95h6m1.570008194s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.130415023 (95h6m1.592507496s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.174952351 (95h6m1.637044824s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.243631279 (95h6m1.705723762s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.339383399 (95h6m1.801475872s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:42.995565076 (95h6m28.457657549s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:58146->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995705543 (95h6m28.457798027s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:37130->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995727205 (95h6m28.457819688s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:42284->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995776930 (95h6m28.457869403s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:56962->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995882301 (95h6m28.457974784s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:55646->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995910995 (95h6m28.458003488s) [seg_pos={0 0}, upload=83668]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 12:37:43.236012986 (95h6m28.698105469s) [upload=83668]: Done uploading segments... Sep 27 12:37:43.236027784 (95h6m28.698120267s) [upload=83668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 16.376397063s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X......X.................................................X..] duration: 6.41481139s FAIL (2025-09-27 12:38:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 12:37:13.690416760 (95h5m59.152509243s) [upload=83668]: Starting upload Sep 27 12:37:13.690515829 (95h5m59.152608312s) [upload=83668]: Done starting upload Sep 27 12:37:13.690666315 (95h5m59.152758808s) [upload=83668]: Uploading segments... Sep 27 12:37:13.690817974 (95h5m59.152910457s) [upload=83668]: Got next segment. Inline: false Sep 27 12:37:13.690869171 (95h5m59.152961654s) [upload=83668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:37:13.690876305 (95h5m59.152968788s) [upload=83668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:37:13.937266910 (95h5m59.399359393s) [seg_pos={0 0}, upload=83668]: Begin upload segment... Sep 27 12:37:13.937340380 (95h5m59.399432863s) [seg_pos={0 0}, upload=83668]: Done begin upload segment. Sep 27 12:37:13.937372270 (95h5m59.399464753s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.937451391 (95h5m59.399543884s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938081951 (95h5m59.400174444s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938563849 (95h5m59.400656332s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938965585 (95h5m59.401058068s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939353333 (95h5m59.401445816s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939739498 (95h5m59.401831981s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940144499 (95h5m59.402236982s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940668768 (95h5m59.402761261s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.942468184 (95h5m59.404560667s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.944294502 (95h5m59.406386985s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.945989398 (95h5m59.408081891s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.947618120 (95h5m59.409710603s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.949213799 (95h5m59.411306282s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.950845245 (95h5m59.412937728s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.952449149 (95h5m59.414541632s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:15.529577113 (95h6m0.991669586s) [upload=83668]: Next returned nil segment Sep 27 12:37:15.529600238 (95h6m0.991692711s) [upload=83668]: Waiting for error group managing segments... Sep 27 12:37:15.909365373 (95h6m1.371457846s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932137335 (95h6m1.394229818s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932507209 (95h6m1.394599692s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.938051066 (95h6m1.400143539s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.941172539 (95h6m1.403265022s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.944182742 (95h6m1.406275225s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.107915721 (95h6m1.570008194s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.130415023 (95h6m1.592507496s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.174952351 (95h6m1.637044824s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.243631279 (95h6m1.705723762s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.339383399 (95h6m1.801475872s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:42.995565076 (95h6m28.457657549s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:58146->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995705543 (95h6m28.457798027s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:37130->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995727205 (95h6m28.457819688s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:42284->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995776930 (95h6m28.457869403s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:56962->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995882301 (95h6m28.457974784s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:55646->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995910995 (95h6m28.458003488s) [seg_pos={0 0}, upload=83668]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 12:37:43.236012986 (95h6m28.698105469s) [upload=83668]: Done uploading segments... Sep 27 12:37:43.236027784 (95h6m28.698120267s) [upload=83668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.414840085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X......X.................................................X.] duration: 6.037601324s FAIL (2025-09-27 12:38:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 12:37:13.690416760 (95h5m59.152509243s) [upload=83668]: Starting upload Sep 27 12:37:13.690515829 (95h5m59.152608312s) [upload=83668]: Done starting upload Sep 27 12:37:13.690666315 (95h5m59.152758808s) [upload=83668]: Uploading segments... Sep 27 12:37:13.690817974 (95h5m59.152910457s) [upload=83668]: Got next segment. Inline: false Sep 27 12:37:13.690869171 (95h5m59.152961654s) [upload=83668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:37:13.690876305 (95h5m59.152968788s) [upload=83668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:37:13.937266910 (95h5m59.399359393s) [seg_pos={0 0}, upload=83668]: Begin upload segment... Sep 27 12:37:13.937340380 (95h5m59.399432863s) [seg_pos={0 0}, upload=83668]: Done begin upload segment. Sep 27 12:37:13.937372270 (95h5m59.399464753s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.937451391 (95h5m59.399543884s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938081951 (95h5m59.400174444s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938563849 (95h5m59.400656332s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938965585 (95h5m59.401058068s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939353333 (95h5m59.401445816s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939739498 (95h5m59.401831981s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940144499 (95h5m59.402236982s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940668768 (95h5m59.402761261s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.942468184 (95h5m59.404560667s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.944294502 (95h5m59.406386985s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.945989398 (95h5m59.408081891s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.947618120 (95h5m59.409710603s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.949213799 (95h5m59.411306282s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.950845245 (95h5m59.412937728s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.952449149 (95h5m59.414541632s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:15.529577113 (95h6m0.991669586s) [upload=83668]: Next returned nil segment Sep 27 12:37:15.529600238 (95h6m0.991692711s) [upload=83668]: Waiting for error group managing segments... Sep 27 12:37:15.909365373 (95h6m1.371457846s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932137335 (95h6m1.394229818s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932507209 (95h6m1.394599692s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.938051066 (95h6m1.400143539s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.941172539 (95h6m1.403265022s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.944182742 (95h6m1.406275225s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.107915721 (95h6m1.570008194s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.130415023 (95h6m1.592507496s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.174952351 (95h6m1.637044824s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.243631279 (95h6m1.705723762s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.339383399 (95h6m1.801475872s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:42.995565076 (95h6m28.457657549s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:58146->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995705543 (95h6m28.457798027s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:37130->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995727205 (95h6m28.457819688s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:42284->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995776930 (95h6m28.457869403s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:56962->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995882301 (95h6m28.457974784s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:55646->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995910995 (95h6m28.458003488s) [seg_pos={0 0}, upload=83668]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 12:37:43.236012986 (95h6m28.698105469s) [upload=83668]: Done uploading segments... Sep 27 12:37:43.236027784 (95h6m28.698120267s) [upload=83668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.037631754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X......X.................................................X] duration: 30.242788464s FAIL (2025-09-27 12:37:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 12:37:13.690416760 (95h5m59.152509243s) [upload=83668]: Starting upload Sep 27 12:37:13.690515829 (95h5m59.152608312s) [upload=83668]: Done starting upload Sep 27 12:37:13.690666315 (95h5m59.152758808s) [upload=83668]: Uploading segments... Sep 27 12:37:13.690817974 (95h5m59.152910457s) [upload=83668]: Got next segment. Inline: false Sep 27 12:37:13.690869171 (95h5m59.152961654s) [upload=83668]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:37:13.690876305 (95h5m59.152968788s) [upload=83668]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:37:13.937266910 (95h5m59.399359393s) [seg_pos={0 0}, upload=83668]: Begin upload segment... Sep 27 12:37:13.937340380 (95h5m59.399432863s) [seg_pos={0 0}, upload=83668]: Done begin upload segment. Sep 27 12:37:13.937372270 (95h5m59.399464753s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.937451391 (95h5m59.399543884s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938081951 (95h5m59.400174444s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938563849 (95h5m59.400656332s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.938965585 (95h5m59.401058068s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939353333 (95h5m59.401445816s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.939739498 (95h5m59.401831981s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940144499 (95h5m59.402236982s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.940668768 (95h5m59.402761261s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.942468184 (95h5m59.404560667s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.944294502 (95h5m59.406386985s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.945989398 (95h5m59.408081891s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.947618120 (95h5m59.409710603s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.949213799 (95h5m59.411306282s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.950845245 (95h5m59.412937728s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:13.952449149 (95h5m59.414541632s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Uploading piece... Sep 27 12:37:15.529577113 (95h6m0.991669586s) [upload=83668]: Next returned nil segment Sep 27 12:37:15.529600238 (95h6m0.991692711s) [upload=83668]: Waiting for error group managing segments... Sep 27 12:37:15.909365373 (95h6m1.371457846s) [noise=true , address=145.79.227.5:4223 , piece_id=VVZT3LYE56Y56IJGWBSYFAAKBBZ4RNSBJABU2TAN6TJ4CCEAPYLA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932137335 (95h6m1.394229818s) [noise=true , address=145.79.227.6:4258 , piece_id=YW2QFS7YBJ4GJ46ZVEX7UWDKURFW4MORWP3P6BVR2YUCHSG4MKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.932507209 (95h6m1.394599692s) [noise=true , address=145.79.227.13:4804 , piece_id=OZCPYCCYCJI5DJLWPKOZRTY76EYVF2SYYRSJHURHY3URP2KJHQ6A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.938051066 (95h6m1.400143539s) [noise=true , address=145.79.227.6:4257 , piece_id=4VQ55Z3C3AHRV7GKC6WQSHSX4ONYWDNW4FRMVHG72KHMRI63AHSA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.941172539 (95h6m1.403265022s) [noise=true , address=145.79.227.14:4837 , piece_id=R7FI6BEZ2N6UBDZ2H3NEVBVEKN4YWICKBAD72WT73GEZKTKB3OCA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:15.944182742 (95h6m1.406275225s) [noise=true , address=145.79.227.16:4912 , piece_id=YELV7SB5V4FZVIQUULBDLT4SXUS7TUWBOKUVWW2K742RIH42TKEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.107915721 (95h6m1.570008194s) [noise=true , address=77.90.31.10:4520 , piece_id=PLPAT2LQQNHBEVTF4ILCSM7MCLLGH5LJLVCEHDUPC2OHL7Z6ZDZQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.130415023 (95h6m1.592507496s) [noise=true , address=77.90.31.13:4628 , piece_id=G5IEE76J4O3QHT4QEXSHGQO66HFKVMNTYBJCJPH3USX5FIF36HVQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.174952351 (95h6m1.637044824s) [noise=true , address=77.90.31.19:4841 , piece_id=2S2WPFZCCSR5OWQQKGGB36B2UFNEIRLXSQGS22BGH6CJ7DYJI2DA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.243631279 (95h6m1.705723762s) [noise=true , address=77.90.31.15:4697 , piece_id=S55EDHCG7S7H6U5QQRAPRO5Q4JBWKHGF66ZG5YUF5ZAOOVCDHZNQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:16.339383399 (95h6m1.801475872s) [noise=true , address=77.90.31.17:4772 , piece_id=SXPFUIVUY4PW4MVDDFGCDGRRXUT5GA7NJUZCGNY4IRA7IQIZLPOQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: <nil> Sep 27 12:37:42.995565076 (95h6m28.457657549s) [noise=true , address=45.91.116.7:4009 , piece_id=XDYOPJQLDUDBC4Y6NYVHSZVGTMDUGT7HEGENR2DUMQQGKFHU3DFA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, address:45.91.116.7:4009): protocol: read tcp 188.166.225.170:58146->45.91.116.7:4009: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995705543 (95h6m28.457798027s) [noise=true , address=45.91.116.4:4938 , piece_id=L5OILLBOWMLO477L3QFMRW4LPUBARSBMMWUP2ROOSYOL4GR7SMEQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:37130->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995727205 (95h6m28.457819688s) [noise=true , address=45.91.116.13:4155 , piece_id=2ATZMT537NLRYVU3DB4JUNJGJVTKOCCEKCKTXA62VLEIGDXRTO7A, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: read tcp 188.166.225.170:42284->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995776930 (95h6m28.457869403s) [noise=true , address=45.91.116.14:4190 , piece_id=QKGKUKCWQJRWQL67ON7I3LRX4CW4HQINAY6YZB6WXR6DI3OWDKTQ, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: read tcp 188.166.225.170:56962->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995882301 (95h6m28.457974784s) [noise=true , address=45.91.116.6:4973 , piece_id=K3MJZAI2UDZRSSGYG5G72N2WJBVXQKQ4IKCBURH2X7JLF5NHR5HA, seg_pos={0 0}, upload=83668]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:55646->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:37:42.995910995 (95h6m28.458003488s) [seg_pos={0 0}, upload=83668]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 12:37:43.236012986 (95h6m28.698105469s) [upload=83668]: Done uploading segments... Sep 27 12:37:43.236027784 (95h6m28.698120267s) [upload=83668]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.242809394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X......X.................................................] duration: 6.084942746s FAIL (2025-09-27 12:36:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.084971091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X......X................................................] duration: 6.287152056s FAIL (2025-09-27 12:35:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.287222409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X......X...............................................] duration: 5.789121711s FAIL (2025-09-27 12:34:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.789153141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X......X..............................................] duration: 11.303732938s FAIL (2025-09-27 12:33:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 11.303761884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X......X.............................................] duration: 6.241927499s FAIL (2025-09-27 12:33:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.241953168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X......X............................................] duration: 6.287514048s FAIL (2025-09-27 12:32:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.287544878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X......X...........................................] duration: 8.392937501s FAIL (2025-09-27 12:31:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 8.39296973s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.........X......X..........................................] duration: 6.496298229s FAIL (2025-09-27 12:31:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 12:01:16.102039370 (94h30m1.564131843s) [upload=83145]: Starting upload Sep 27 12:01:16.102132427 (94h30m1.56422491s) [upload=83145]: Done starting upload Sep 27 12:01:16.102198192 (94h30m1.564290665s) [upload=83145]: Uploading segments... Sep 27 12:01:16.102344310 (94h30m1.564436793s) [upload=83145]: Got next segment. Inline: false Sep 27 12:01:16.102378255 (94h30m1.564470738s) [upload=83145]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 12:01:16.102383124 (94h30m1.564475608s) [upload=83145]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 12:01:16.365619528 (94h30m1.827712021s) [seg_pos={0 0}, upload=83145]: Begin upload segment... Sep 27 12:01:16.365704430 (94h30m1.827796913s) [seg_pos={0 0}, upload=83145]: Done begin upload segment. Sep 27 12:01:16.365839626 (94h30m1.82793213s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370001872 (94h30m1.832094356s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370028303 (94h30m1.832120786s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370176255 (94h30m1.832268738s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370701595 (94h30m1.832794078s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.370711995 (94h30m1.832804478s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371170488 (94h30m1.833262971s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371397360 (94h30m1.833489843s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.371597922 (94h30m1.833690405s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372025486 (94h30m1.834117959s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372125777 (94h30m1.83421826s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372702355 (94h30m1.834794858s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.372715991 (94h30m1.834808474s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373333938 (94h30m1.835426421s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.373998433 (94h30m1.836090916s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:16.374632671 (94h30m1.836725144s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Uploading piece... Sep 27 12:01:18.072094304 (94h30m3.534186787s) [upload=83145]: Next returned nil segment Sep 27 12:01:18.072117429 (94h30m3.534209912s) [upload=83145]: Waiting for error group managing segments... Sep 27 12:01:18.359880587 (94h30m3.82197307s) [noise=true , address=145.79.227.6:4259 , piece_id=NT3E2NAUPGUPRI2ZJWKGH3LX2NUOE7VLZKHWMOHOUN72MWL5YR7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.365228130 (94h30m3.827320613s) [noise=true , address=145.79.227.7:4293 , piece_id=647V5CFJQ4E7MA4XPZB74CYSYNIBATESTRKVCSF6VKFQGG6HK36A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.380529778 (94h30m3.842622261s) [noise=true , address=145.79.227.17:4971 , piece_id=MA5BNFNDPEEWAHJETEAEY5CYSMH47ATWDHCG575YONL6KD5JXFSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.597348920 (94h30m4.059441403s) [noise=true , address=45.91.116.6:4974 , piece_id=QUZW2MR2KJ2VQRRDWP2YTPIFZ2JWXDV6JVW3ZWVZK7PUZCUZZFRA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.607552301 (94h30m4.069644774s) [noise=true , address=45.91.116.7:4010 , piece_id=PNLY7EBP5QTPIFPT6HJSJBJYMWNW3WQ7BOHOH7W7YCSQJ6TZUASA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.701217024 (94h30m4.163309497s) [noise=true , address=45.91.116.13:4153 , piece_id=SH5O6RLVKOQVXCRF7LGKHEO2XYWF32CTUYGAON4OTHN66E6U7LOQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.705498987 (94h30m4.167591471s) [noise=true , address=45.91.116.4:4939 , piece_id=XMIKHPUWDQB6QRFNHYTDVYVY2B6DUZASDWGVD2MH2W2BAO4SZNAA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:18.831093122 (94h30m4.293185606s) [noise=true , address=45.91.116.8:4048 , piece_id=VV2WPBIFPEN3ABOQGSLD5BULTVZ2YIL2S2LKIWZSLIH5ZDJCDSUA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.017939018 (94h30m4.480031491s) [noise=true , address=45.91.116.5:4299 , piece_id=R6O4WZHYPQL4GV36W5SMVOY5BJEPS3RHPZNUP67HI2CEXN7VTM7A, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:19.214667261 (94h30m4.676759744s) [noise=true , address=45.91.116.11:4118 , piece_id=RKCWK5IGSEHZ6RAN4WJFIBBM4YP2TQFZV6EE43HODBQU6ROXPMSQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: <nil> Sep 27 12:01:45.363407832 (94h30m30.825500315s) [noise=true , address=77.90.31.18:4807 , piece_id=DRZJ4JPKSE2EPGCE44MNQO6F7YQBEVUQVAME6BHWHHHIVMMGNOGQ, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363647478 (94h30m30.825739971s) [noise=true , address=77.90.31.15:4698 , piece_id=7KVB72VOXMB57LCBYSZXCUR63OEXZS4MQUME7LIEMCT7FWOSQO5Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.363856556 (94h30m30.825949039s) [noise=true , address=77.90.31.16:4735 , piece_id=HRLDPLEGYVSCA2OEGK3KAWDSEW3LCLY35LMGRGRVHX6ZVXR3DUJA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: rpc: dial tcp 77.90.31.16:4735: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364092496 (94h30m30.826184979s) [noise=true , address=77.90.31.15:4697 , piece_id=UPZFEVEAXZ7GVAMLE3FLGS2L5CAXNHXTQZ7IWOCJZ47OJTBCCH3Q, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: rpc: dial tcp 77.90.31.15:4697: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364307605 (94h30m30.826400088s) [noise=true , address=77.90.31.17:4770 , piece_id=KSWXSF3L6WBS5MMEZPGV32YQH5H5VQK2DW747AA4K7QK7CEG2ISA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364510912 (94h30m30.826603395s) [noise=true , address=77.90.31.14:4663 , piece_id=VD7PZIQNMGDPTT7R5SGQGLBJMHC3FGNET22NFUDI6LG35YEYWFTA, seg_pos={0 0}, upload=83145]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35690->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 12:01:45.364549756 (94h30m30.826642239s) [seg_pos={0 0}, upload=83145]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 12:01:45.617281936 (94h30m31.079374429s) [upload=83145]: Done uploading segments... Sep 27 12:01:45.617318986 (94h30m31.079411479s) [upload=83145]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.496328126s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)