Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 23.597735282s FAIL (2025-09-27 11:03:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 23.597779967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X..........X.................................X..] duration: 10.825563842s FAIL (2025-09-27 11:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.825596144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.40854646s FAIL (2025-09-27 11:02:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.408578201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.970078672s FAIL (2025-09-27 11:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.970109662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X..........X.................................X.] duration: 6.143897193s FAIL (2025-09-27 11:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.143932059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X..........X.................................X] duration: 30.26792831s FAIL (2025-09-27 11:00:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.267957345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.825109169s FAIL (2025-09-27 11:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.825145688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.X..........X.................................] duration: 11.507961605s FAIL (2025-09-27 11:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.507997183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.092091676s FAIL (2025-09-27 10:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.092124139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.X..........X................................] duration: 7.752448886s FAIL (2025-09-27 10:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.752476228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.948225021s FAIL (2025-09-27 10:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.948257063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.744309594s FAIL (2025-09-27 10:58:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.744369939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.X..........X...............................] duration: 6.079818551s FAIL (2025-09-27 10:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.079859058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.X..........X..............................] duration: 6.314750022s FAIL (2025-09-27 10:57:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.314782463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.283171012s FAIL (2025-09-27 10:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 7.28320101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.X..........X.............................] duration: 7.436803356s FAIL (2025-09-27 10:56:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.436830829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.300051633s FAIL (2025-09-27 10:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.300083455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.703128957s FAIL (2025-09-27 10:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.703160659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X..........X............................] duration: 6.889821428s FAIL (2025-09-27 10:55:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.889860652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X..........X...........................] duration: 6.179314298s FAIL (2025-09-27 10:55:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.179348964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.89074801s FAIL (2025-09-27 10:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.890779871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.900344742s FAIL (2025-09-27 10:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.900383795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X..........X..........................] duration: 6.168689948s FAIL (2025-09-27 10:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.168721508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.254154487s FAIL (2025-09-27 10:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.254189993s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.492690995s FAIL (2025-09-27 10:53:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.49272554s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.X..........X.........................] duration: 6.091185594s FAIL (2025-09-27 10:53:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.091221834s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.46131182s FAIL (2025-09-27 10:52:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.46134247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.X..........X........................] duration: 7.206644615s FAIL (2025-09-27 10:51:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.206682246s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.197885381s FAIL (2025-09-27 10:51:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.197919435s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X......................X.X..........X.......................] duration: 7.183711616s FAIL (2025-09-27 10:51:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.183743335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X......................X.X..........X......................] duration: 7.414648309s FAIL (2025-09-27 10:50:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.414679528s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..X......................X.X..........X.....................] duration: 6.980822197s FAIL (2025-09-27 10:50:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.980864778s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 12.632117425s FAIL (2025-09-27 10:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 12.632157621s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 8.430847068s FAIL (2025-09-27 10:49:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 8.430884528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X......................X.X..........X....................] duration: 7.252681564s FAIL (2025-09-27 10:49:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.252713585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 21.962254989s FAIL (2025-09-27 10:49:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 21.962283725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X......................X.X..........X...................] duration: 6.138827724s FAIL (2025-09-27 10:48:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.138871487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X......................X.X..........X..................] duration: 7.160286849s FAIL (2025-09-27 10:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.160321455s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.835109564s FAIL (2025-09-27 10:47:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.835137117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.135619555s FAIL (2025-09-27 10:47:38 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.135650786s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X......................X.X..........X.................] duration: 6.20563565s FAIL (2025-09-27 10:47:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.205670496s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 7.423213367s FAIL (2025-09-27 10:46:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 7.423250027s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X......................X.X..........X................] duration: 6.681633901s FAIL (2025-09-27 10:46:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.681666243s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.888813434s FAIL (2025-09-27 10:46:43 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 5.888844263s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.852114382s FAIL (2025-09-27 10:46:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.852147445s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.374309783s FAIL (2025-09-27 10:46:06 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.374366972s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X......................X.X..........X...............] duration: 5.920912656s FAIL (2025-09-27 10:45:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.920946591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.053462505s FAIL (2025-09-27 10:45:34 UTC) orbiter warning: operation: sync2.Copy uplink: encryption: failed to upload enough pieces (needed at least 49 but got 0) Duration: 6.053495817s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.822851801s FAIL (2025-09-27 10:45:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.822879424s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X......................X.X..........X..............] duration: 6.265892869s FAIL (2025-09-27 10:45:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.265926373s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)