Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X..........X.................................X....X.........] duration: 6.125808741s FAIL (2025-09-27 11:13:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.125842026s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 9.601054445s FAIL (2025-09-27 11:13:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 9.601081286s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X..........X.................................X....X........] duration: 5.971670184s FAIL (2025-09-27 11:12:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.971700984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.208725046s FAIL (2025-09-27 11:12:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.208746597s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 38.516926307s FAIL (2025-09-27 11:12:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 38.516957384s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X..........X.................................X....X.......] duration: 6.211813346s FAIL (2025-09-27 11:11:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.211840319s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.483826316s FAIL (2025-09-27 11:11:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.483885128s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X..........X.................................X....X......] duration: 23.715835255s FAIL (2025-09-27 11:10:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 23.715870011s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.24339298s FAIL (2025-09-27 11:10:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.243427987s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X..........X.................................X....X.....] duration: 6.144935701s FAIL (2025-09-27 11:09:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.144966531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.189293988s FAIL (2025-09-27 11:09:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.18933196s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.507189125s FAIL (2025-09-27 11:09:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.507214322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X..........X.................................X....X....] duration: 6.052149361s FAIL (2025-09-27 11:08:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.052177785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.63684885s FAIL (2025-09-27 11:08:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.636884528s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X..........X.................................X....X...] duration: 7.006220646s FAIL (2025-09-27 11:07:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.006269418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.157988097s FAIL (2025-09-27 11:07:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.158020158s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X..........X.................................X....X..] duration: 11.768614163s FAIL (2025-09-27 11:07:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.768678386s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.979299022s FAIL (2025-09-27 11:06:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.979333427s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.230344974s FAIL (2025-09-27 11:06:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.230377344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 49.366272566s FAIL (2025-09-27 11:05:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 49.366305337s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X.X..........X.................................X....X.] duration: 6.091924931s FAIL (2025-09-27 11:05:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.091956531s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X.X..........X.................................X....X] duration: 30.247457762s FAIL (2025-09-27 11:05:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.247569625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X.X..........X.................................X....] duration: 8.396860216s FAIL (2025-09-27 11:04:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 8.396893861s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.880648022s FAIL (2025-09-27 11:04:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.880677309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 11.793490488s FAIL (2025-09-27 11:03:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 11.793519394s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X.X..........X.................................X...] duration: 6.380068026s FAIL (2025-09-27 11:03:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.380096869s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 23.597735282s FAIL (2025-09-27 11:03:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 23.597779967s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..........X.X..........X.................................X..] duration: 10.825563842s FAIL (2025-09-27 11:02:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 10.825596144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.40854646s FAIL (2025-09-27 11:02:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.408578201s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.970078672s FAIL (2025-09-27 11:01:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.970109662s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...........X.X..........X.................................X.] duration: 6.143897193s FAIL (2025-09-27 11:01:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.143932059s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [............X.X..........X.................................X] duration: 30.26792831s FAIL (2025-09-27 11:00:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:00:29.718239715 (93h29m15.180332198s) [upload=82257]: Starting upload Sep 27 11:00:29.718320950 (93h29m15.180413433s) [upload=82257]: Done starting upload Sep 27 11:00:29.718538114 (93h29m15.180630607s) [upload=82257]: Uploading segments... Sep 27 11:00:29.718666889 (93h29m15.180759372s) [upload=82257]: Got next segment. Inline: false Sep 27 11:00:29.718710252 (93h29m15.180802725s) [upload=82257]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:00:29.718717205 (93h29m15.180809688s) [upload=82257]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:00:29.991756137 (93h29m15.45384863s) [seg_pos={0 0}, upload=82257]: Begin upload segment... Sep 27 11:00:29.991834205 (93h29m15.453926688s) [seg_pos={0 0}, upload=82257]: Done begin upload segment. Sep 27 11:00:29.991894801 (93h29m15.453987284s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.991914549 (93h29m15.454007032s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992032092 (93h29m15.454124575s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992629029 (93h29m15.454721512s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.992852675 (93h29m15.454945148s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.993462786 (93h29m15.455555269s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994243894 (93h29m15.456336377s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.994979004 (93h29m15.457071497s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996003895 (93h29m15.458096378s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.996442911 (93h29m15.458535395s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997740833 (93h29m15.459833316s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.997789194 (93h29m15.459881677s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999160456 (93h29m15.461252949s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:29.999338305 (93h29m15.461430798s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.000164248 (93h29m15.462256741s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:30.002479816 (93h29m15.464572319s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Uploading piece... Sep 27 11:00:31.707854029 (93h29m17.169946512s) [upload=82257]: Next returned nil segment Sep 27 11:00:31.707881682 (93h29m17.169974165s) [upload=82257]: Waiting for error group managing segments... Sep 27 11:00:31.990884408 (93h29m17.452976891s) [noise=true , address=145.79.227.2:4113 , piece_id=LWIPSTIY3U6DBVGDDETQZV5XNOFFGWGXBEAO5DFR7NQEIXIHXHEQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.005159770 (93h29m17.467252253s) [noise=true , address=145.79.227.15:4875 , piece_id=ECINYDVS3A747D2OMVUIQ6NYUNFXISMCXNOWWVMUH7WPWGRDNXAQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.022700020 (93h29m17.484792503s) [noise=true , address=145.79.227.17:4971 , piece_id=QDR3PI72QGR2R6QZIIQK5D7RNANHUCCOV6YY3HAD6RQSBC3VOIGQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.158450432 (93h29m17.620542915s) [noise=true , address=45.91.116.4:4938 , piece_id=G64FT7ZEYM2Q7F3OVUPRTDKSXANQXGLPD77EMQABYZVA4O6JKRRA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.171761778 (93h29m17.633854261s) [noise=true , address=45.91.116.6:4973 , piece_id=SUDQKBAX4QOZD5MQEA7KJE4AA5GUYCB6Y47RXG234675AX53VW5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.183523485 (93h29m17.645615978s) [noise=true , address=45.91.116.8:4045 , piece_id=IJ7IDWPNHEJKXDN4SHMRJOXT2ZQU3AACHH2FBZTLATQNFP2OADZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.190520498 (93h29m17.652612991s) [noise=true , address=45.91.116.4:4939 , piece_id=REZDPYTR42UBTXNRZU3QE2U3ZODL2FF4VBGCUPA6PEP47GLZ4YEA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.193385795 (93h29m17.655478278s) [noise=true , address=45.91.116.12:4227 , piece_id=PEHDTKXQY3CE4L5JKFG5RHXJEIUNUJSMSDHQPUONZDA74BOURB5Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.216096860 (93h29m17.678189343s) [noise=true , address=45.91.116.10:4083 , piece_id=AXVL37WF3TSFXSG6HQOJFXMCJK7KRCKL4Z3U5FULQHHFVD3HN5BQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.232303351 (93h29m17.694395834s) [noise=true , address=45.91.116.10:4081 , piece_id=2QCF4MEAMBNXY67E42KY7DB4TE337F6PTHTB3PFUWUK267SJBYUA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:32.267852222 (93h29m17.729944705s) [noise=true , address=45.91.116.14:4190 , piece_id=YRWSTY6SWWMMROLH7JE3MKDSYFRGHCO6LOVNVU7QC56V6J6ZRNHQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: <nil> Sep 27 11:00:58.944068409 (93h29m44.406160892s) [noise=true , address=77.90.31.15:4697 , piece_id=L5T3H4NZCNW2DJ4YLD5IWVPHMG6JIYUH2CURQJOOP2MGHAKTS5DA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:55218->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944185291 (93h29m44.406277764s) [noise=true , address=77.90.31.10:4518 , piece_id=S2HBQSDZDVH4QYGGZWC4HGJLJIXASB776VHGLFQH55XKTXCO6ZWA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: write tcp 188.166.225.170:50364->77.90.31.10:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944216872 (93h29m44.406309345s) [noise=true , address=77.90.31.10:4517 , piece_id=SR5RL7J7WI7PCPBGMYSZRYX572HVWTDWCPIJXFMHBPMR2W76PPZA, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:58658->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944235738 (93h29m44.406328211s) [noise=true , address=77.90.31.16:4735 , piece_id=G2MFQBS6RLHCWKOWEN2G3S6XLENGKZ2LGUINWFEYNHS33TOTNQZQ, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, address:77.90.31.16:4735): protocol: write tcp 188.166.225.170:48374->77.90.31.16:4735: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:00:58.944391113 (93h29m44.406483597s) [noise=true , address=77.90.31.12:4590 , piece_id=U7BQDG55GXBWCI5ZD5LULX5EMPPJKWQTNPCAMUXAC5Y45UNC327Q, seg_pos={0 0}, upload=82257]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: write tcp 188.166.225.170:59668->77.90.31.12:4590: use of closed network connection; context deadline exceeded Sep 27 11:00:58.944418666 (93h29m44.406511139s) [seg_pos={0 0}, upload=82257]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:00:59.210737253 (93h29m44.672829736s) [upload=82257]: Done uploading segments... Sep 27 11:00:59.210768102 (93h29m44.672860575s) [upload=82257]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.267957345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.825109169s FAIL (2025-09-27 11:00:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.825145688s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.X..........X.................................] duration: 11.507961605s FAIL (2025-09-27 11:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.507997183s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.092091676s FAIL (2025-09-27 10:59:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.092124139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.X..........X................................] duration: 7.752448886s FAIL (2025-09-27 10:59:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.752476228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.948225021s FAIL (2025-09-27 10:58:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.948257063s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.744309594s FAIL (2025-09-27 10:58:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.744369939s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.X..........X...............................] duration: 6.079818551s FAIL (2025-09-27 10:58:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.079859058s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.X..........X..............................] duration: 6.314750022s FAIL (2025-09-27 10:57:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.314782463s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.283171012s FAIL (2025-09-27 10:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 7.28320101s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.X..........X.............................] duration: 7.436803356s FAIL (2025-09-27 10:56:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.436830829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.300051633s FAIL (2025-09-27 10:56:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.300083455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.703128957s FAIL (2025-09-27 10:55:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.703160659s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.X..........X............................] duration: 6.889821428s FAIL (2025-09-27 10:55:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.889860652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.X..........X...........................] duration: 6.179314298s FAIL (2025-09-27 10:55:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.179348964s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.89074801s FAIL (2025-09-27 10:55:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.890779871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.900344742s FAIL (2025-09-27 10:54:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 5.900383795s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.X..........X..........................] duration: 6.168689948s FAIL (2025-09-27 10:54:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 10:35:55.492618801 (93h4m40.954711284s) [upload=81895]: Starting upload Sep 27 10:35:55.492724953 (93h4m40.954817436s) [upload=81895]: Done starting upload Sep 27 10:35:55.492961173 (93h4m40.955053656s) [upload=81895]: Uploading segments... Sep 27 10:35:55.493102983 (93h4m40.955195466s) [upload=81895]: Got next segment. Inline: false Sep 27 10:35:55.493143730 (93h4m40.955236214s) [upload=81895]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 10:35:55.493151255 (93h4m40.955243738s) [upload=81895]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 10:35:55.801253905 (93h4m41.263346388s) [seg_pos={0 0}, upload=81895]: Begin upload segment... Sep 27 10:35:55.801318629 (93h4m41.263411102s) [seg_pos={0 0}, upload=81895]: Done begin upload segment. Sep 27 10:35:55.801347865 (93h4m41.263440348s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801364066 (93h4m41.263456539s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801882032 (93h4m41.263974515s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.801996771 (93h4m41.264089254s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802312892 (93h4m41.264405376s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802647330 (93h4m41.264739803s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.802900722 (93h4m41.264993195s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803138785 (93h4m41.265231258s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803393991 (93h4m41.265486464s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803657173 (93h4m41.265749656s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.803900837 (93h4m41.26599332s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804200657 (93h4m41.26629314s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804465752 (93h4m41.266558235s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.804786162 (93h4m41.266878635s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.805185963 (93h4m41.267278446s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:55.806234761 (93h4m41.268327244s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Uploading piece... Sep 27 10:35:57.521833510 (93h4m42.983925993s) [upload=81895]: Next returned nil segment Sep 27 10:35:57.521869569 (93h4m42.983962052s) [upload=81895]: Waiting for error group managing segments... Sep 27 10:35:57.803163881 (93h4m43.265256364s) [noise=true , address=145.79.227.6:4258 , piece_id=HBIGQWJ7IZY7HBUVH2XUUXNO7UAME5IBZ4SDDY6IX7EOEVEVCOXA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808216382 (93h4m43.270308865s) [noise=true , address=145.79.227.18:4006 , piece_id=EKMREABYHJXPFWFYCXOQQ3PKEVUP4KSNTFKXKUIZ24255ZY3PQSQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.808425279 (93h4m43.270517762s) [noise=true , address=145.79.227.13:4802 , piece_id=E57P6WC5E5E3QA5JQSS7LEGCAV2Q436JE5JYXD7MWQF3GPSGYSQA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.809242215 (93h4m43.271334688s) [noise=true , address=145.79.227.5:4222 , piece_id=RCVOA3MW4DVRQDEJFIIOHXMBN4TJSS3E33KC2V2QZRZRO5PKLJJA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.810175221 (93h4m43.272267704s) [noise=true , address=145.79.227.15:4874 , piece_id=237OK5GORGBX4AMJEFNI3KRP4WHVRMNP3HBID3375ABMHM2JZKVQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.811431825 (93h4m43.273524308s) [noise=true , address=145.79.227.17:4972 , piece_id=IWXMOEC442ABYSA6KMIOIG4TEQJUHHDL34VUWPD4GPXSJQVHD7EQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.985489760 (93h4m43.447582243s) [noise=true , address=45.91.116.12:4225 , piece_id=CMJ5DEPDU2WJALCWOWNOK7SWJEUEHXCO45G5YOE7LS6EH3SOTH4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:57.993106054 (93h4m43.455198537s) [noise=true , address=45.91.116.7:4010 , piece_id=OXA6EMMM6MDZCRVCDFPXFI5SRYBZNT3HQ5O5ADIIMSMH65WRSJTA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.002470336 (93h4m43.464562829s) [noise=true , address=45.91.116.10:4083 , piece_id=HMOC6BWVHK2U5UQ72WGIOZ5RSB2OQB7SEHV7XGU3RZLQR3B34D4A, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016048772 (93h4m43.478141255s) [noise=true , address=45.91.116.5:4298 , piece_id=LQWBXXJTO3FKT2MYKWZFXUWEJSNNP5AJP5STD73L5M7QZJWGDYOA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.016357270 (93h4m43.478449753s) [noise=true , address=45.91.116.6:4973 , piece_id=NAMDEVOXKXMXGAF5J7FX2Q3OUZYEILXA5OVNP2CZVWOE4CMONM4Q, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:35:58.042820913 (93h4m43.504913396s) [noise=true , address=45.91.116.6:4975 , piece_id=5OH7VUOXCLG4YWJ5SOYX77BS34P74M4HKSMSPY64QNQKLY2MISZQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: <nil> Sep 27 10:36:24.774918739 (93h5m10.237011232s) [noise=true , address=77.90.31.21:4877 , piece_id=CLEJZ3UFUFMAJT2QEHDNZGLVRLYQV52LSKLXAYTG3WDSMTF5YYMA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:59084->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.774985846 (93h5m10.237078329s) [noise=true , address=77.90.31.10:4520 , piece_id=V2437TY4L44WIF25PYOIIMG46THUTKBNAKI4XDBSEMM7VYLM4YKQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:60078->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775185727 (93h5m10.23727821s) [noise=true , address=77.90.31.13:4628 , piece_id=4GI5I6WV4B7GVNC4NLD3RDBKAYUXRVQL7FTRT4KHELQER6UJNTFA, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ERDD2hvVwWnBGdG7C1aCWMZGzbGHtKeKio5fwRkmo5f8fms8a, address:77.90.31.13:4628): protocol: rpc: dial tcp 77.90.31.13:4628: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775247985 (93h5m10.237340468s) [noise=true , address=77.90.31.11:4553 , piece_id=OGUDRCZSO7XLFXKJ4D47KXVUOGKQ67MFJPGJZ333RHUD6HNAXNTQ, seg_pos={0 0}, upload=81895]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: write tcp 188.166.225.170:55010->77.90.31.11:4553: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 10:36:24.775281559 (93h5m10.237374042s) [seg_pos={0 0}, upload=81895]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 10:36:25.046682221 (93h5m10.508774704s) [upload=81895]: Done uploading segments... Sep 27 10:36:25.046710814 (93h5m10.508803297s) [upload=81895]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.168721508s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.254154487s FAIL (2025-09-27 10:54:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: eestream: quiescence ======LOGS====== 77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278251929 (93h8m36.740344412s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278338142 (93h8m36.740430616s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278609008 (93h8m36.740701492s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.278981238 (93h8m36.741073711s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279003740 (93h8m36.741096223s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279347555 (93h8m36.741440038s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279444319 (93h8m36.741536792s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279695307 (93h8m36.74178779s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.279922079 (93h8m36.742014552s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280099928 (93h8m36.742192411s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280367037 (93h8m36.74245952s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280472798 (93h8m36.742565271s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280887267 (93h8m36.74297975s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.280904950 (93h8m36.742997433s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281278442 (93h8m36.743370925s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.281487400 (93h8m36.743579883s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:51.282814106 (93h8m36.744906669s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Uploading piece... Sep 27 10:39:52.740863449 (93h8m38.202955932s) [upload=81952]: Next returned nil segment Sep 27 10:39:52.740888266 (93h8m38.202980749s) [upload=81952]: Waiting for error group managing segments... Sep 27 10:39:53.140296204 (93h8m38.602388677s) [noise=true , address=145.79.227.47:4279 , piece_id=MGAZ7YKKXKSK2OWNGSPIATDJ7HWCKNDKBMN7MQBG57XB7APGUBBQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145533337 (93h8m38.60762581s) [noise=true , address=145.79.227.34:4926 , piece_id=YSH4PGI6TA2L4V2P4HIYU3K5VEW5N3IPOW5AUIS7R7ZXHO22RNWQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.145707398 (93h8m38.607799881s) [noise=true , address=145.79.227.5:4228 , piece_id=C63TRZDFRX6XTBAWT4RSX7YYJJVBWXL5WNPSXX6FUKGTVIBWDHUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.150888624 (93h8m38.612981107s) [noise=true , address=145.79.227.35:4503 , piece_id=LC3RJFY4GHNHVZ2ZYIINEKT4CG7PPEC66GOXWLYQYTOQACC2S7WA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.152686027 (93h8m38.6147785s) [noise=true , address=145.79.227.49:4362 , piece_id=SDOTXQ3K5XSGU4ABBSNG2IQQ3D5ER4SANF5PANKGMYQZAFKWGDEQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.165625016 (93h8m38.627717489s) [noise=true , address=145.79.227.41:4106 , piece_id=2WCGW2WGELY3EJ3IUHAIWWZIIWIL7D5AVBV36WNFQHKDBKTWZH4Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.180170614 (93h8m38.642263097s) [noise=true , address=145.79.227.29:4945 , piece_id=R73SQKLWT52RC4ZZ5LK46WNUUN4VLO7R5D77Z2HQA25BFLQJMAUQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.203914357 (93h8m38.66600684s) [noise=true , address=145.79.227.20:4440 , piece_id=P5VWOIBCOEQS2O47EVPGXXFWAPCAH2GIF55D3IK6EAQ57LMF4GVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.259462875 (93h8m38.721555359s) [noise=true , address=45.91.116.4:4962 , piece_id=IARE24GFJQOTPKV47QGB743WMATS3OLHMK3LXNOOKNGYVHI7WSZQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.320030240 (93h8m38.782122713s) [noise=true , address=45.91.116.6:4987 , piece_id=Y7BFCHKMOKZAII5SLQG4ZOZZBZK6PINWK7YLMAUWNOXOIIXM2XXA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.330156412 (93h8m38.792248885s) [noise=true , address=45.91.116.11:4142 , piece_id=OXBJ7FKL365QVPLCVJY3BJ7NHNTCDCGSOPEH5JTGACLYFVIJRB3Q, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.339396758 (93h8m38.801489231s) [noise=true , address=77.90.31.11:4563 , piece_id=RUGNATGMZ24CMUBAO56RNO3DGIKPTMBSCDBM6Y5Q6RSHHGPXPBHQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.355967122 (93h8m38.818059605s) [noise=true , address=77.90.31.17:4780 , piece_id=65SJW7UPSNRKUNGK55PECED2Q5YVU7PN5M6BKHGOFH7E5FEAJZVQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415462236 (93h8m38.877554719s) [noise=true , address=77.90.31.18:4821 , piece_id=3BRBNGNXO4Y2LVNKYZZDO6XKQ2KH5LSA7T7U4GUJYTGMJ2ZIKBQQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: <nil> Sep 27 10:39:53.415500599 (93h8m38.877593082s) [seg_pos={0 0}, upload=81952]: Segment reached optimal threshold of 14 pieces. Sep 27 10:39:53.415654071 (93h8m38.877746554s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EWyjRCErG9odC8xXPFaLXNWfsega2SLNbKHjbawahraGqDCwr): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415666163 (93h8m38.877758636s) [noise=true , address=77.90.31.10:4551 , piece_id=HVTYZO3CVOIX3OMIMBZZDQ5P2DDZCGCNKTHTS2SN6VVCJHR7ZALQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415751977 (93h8m38.87784446s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415759672 (93h8m38.877852155s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AgtYnGt6M45rGyTZRr4qCFp1BEkgNPBxxsWqBKEn1Cx7AbYLu): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415783708 (93h8m38.877876181s) [noise=true , address=77.90.31.12:4608 , piece_id=PN3FKWQUAZUR3CRXJIW432OLA6MQAWF4PH4BKGKWORH27UJUB3ZA, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415814186 (93h8m38.877906749s) [noise=true , address=77.90.31.13:4630 , piece_id=BTGIHZM3WJV5LIXNHNH46YK5WXGXQGI3K3PL3A7H5CBJB64X4C3A, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415822041 (93h8m38.877914524s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ysgMdZQcnmzQ5Vfv3RadZN8widTZKrqDnDSthxkYEUXKgRxxF): protocol: expected piece hash; context canceled; EOF Sep 27 10:39:53.415830557 (93h8m38.87792303s) [noise=true , address=77.90.31.19:4845 , piece_id=5W6LGOIIK6YPVH5MGO5WJ3RWHQ3NKQEQRFMZI4225EKFEDNFJ7CQ, seg_pos={0 0}, upload=81952]: Long tail cancellation due to optimal threshold Sep 27 10:39:53.415852278 (93h8m38.877944751s) [seg_pos={0 0}, upload=81952]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Sep 27 10:39:53.415946387 (93h8m38.87803886s) [upload=81952]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 27 10:39:53.415953852 (93h8m38.878046335s) [upload=81952]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 27 10:39:53.746816422 (93h8m39.208908905s) [upload=81952]: Done uploading segments... Sep 27 10:39:53.746840508 (93h8m39.208932991s) [upload=81952]: Upload finished. err: <nil> Duration: 6.254189993s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)