Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.707520839s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:47:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.707549885s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.665324479s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:46:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.66535584s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X........X...X........XX........X........................] duration: 5.929965689s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:46:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.929992429s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.632805996s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:46:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.632829751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X........X...X........XX........X.......................] duration: 5.774280045s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:45:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.77430931s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.663060927s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:45:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.663119208s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X....X........X...X........XX........X......................] duration: 6.011162458s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:45:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.011180672s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.716956825s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:44:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.716986684s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X........X...X........XX........X.....................] duration: 6.119884953s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:44:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.119916081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.818646373s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:43:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.81867183s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X....X........X...X........XX........X....................] duration: 5.981829226s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.981855396s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X....X........X...X........XX........X...................] duration: 5.893858727s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.893883966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.689864728s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:43:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.6898919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X....X........X...X........XX........X..................] duration: 5.94133304s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:43:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.941367695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.758661412s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:42:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.758750932s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.....X....X........X...X........XX........X.................] duration: 5.950495671s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:41:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.950522601s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [......X....X........X...X........XX........X................] duration: 6.010898554s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:41:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.010921497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.......X....X........X...X........XX........X...............] duration: 5.829882747s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.829911562s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.833934793s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:40:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.833961383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [........X....X........X...X........XX........X..............] duration: 5.944733299s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.94476047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.837364856s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:40:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.837451901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........X....X........X...X........XX........X.............] duration: 5.940572847s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:40:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.940604726s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........X....X........X...X........XX........X............] duration: 6.072612941s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:39:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.072639661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.838187137s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:39:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.838228055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........X....X........X...X........XX........X...........] duration: 5.887867835s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:38:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.887897392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.825728727s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:38:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.825773151s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............X....X........X...X........XX........X..........] duration: 5.863102253s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:38:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.863132379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.824287585s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:37:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.824314667s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X............X....X........X...X........XX........X.........] duration: 5.913406314s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:37:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.913434258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.638958344s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.638989172s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.809120591s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.809148594s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX............X....X........X...X........XX........X........] duration: 5.992651615s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:36:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.992673847s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.837629515s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:36:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.837657549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX............X....X........X...X........XX........X.......] duration: 5.920878687s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:36:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.920902913s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.745627843s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:35:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.745656698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX............X....X........X...X........XX........X......] duration: 5.897003658s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:35:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.897035399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX............X....X........X...X........XX........X.....] duration: 5.839899676s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:34:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.839918312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.720924323s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:34:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.720951224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.764240078s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:33:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.76427215s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX............X....X........X...X........XX........X....] duration: 5.857596256s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:33:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.857627565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XX............X....X........X...X........XX........X...] duration: 5.907725426s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:33:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 5.907753751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.573644141s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:32:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.573677044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......XX............X....X........X...X........XX........X..] duration: 6.103746582s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:32:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.103768915s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.790383781s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:31:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.790410692s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......XX............X....X........X...X........XX........X.] duration: 6.066387749s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:31:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 6.066418337s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.650393916s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:31:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.650419255s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........XX............X....X........X...X........XX........X] duration: 30.248786593s trace: https://jaeger.dev.storj.io/trace/7dcd1538a23f4daf FAIL (2025-11-13 23:31:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== s) [seg_pos={0 0}, upload=98046]: Done begin upload segment. Nov 13 23:30:38.240771858 (111h26m36.792242319s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.241980429 (111h26m36.79345089s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242422170 (111h26m36.793892631s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.242828284 (111h26m36.794298745s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243258152 (111h26m36.794728603s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.243669305 (111h26m36.795139766s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244220114 (111h26m36.795690575s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.244628111 (111h26m36.796098572s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245088177 (111h26m36.796558628s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.245557760 (111h26m36.797028212s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.247011018 (111h26m36.798481479s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.248670939 (111h26m36.80014139s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.250524168 (111h26m36.801994619s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.251402650 (111h26m36.802873111s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.255683462 (111h26m36.807153923s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:38.257595654 (111h26m36.809066115s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Uploading piece... Nov 13 23:30:39.998019525 (111h26m38.549489986s) [upload=98046]: Next returned nil segment Nov 13 23:30:39.998035004 (111h26m38.549505455s) [upload=98046]: Waiting for error group managing segments... Nov 13 23:30:40.324830156 (111h26m38.876300617s) [noise=true , address=45.91.116.11:4118 , piece_id=XOKSZL5CLT2PGCPVYLEPJRUW7CCH2XSJ3JB3OGYH7QKOXEA6VWTQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.347399583 (111h26m38.898870034s) [noise=true , address=77.90.31.12:4590 , piece_id=JKIQL6GRMVSI3R56T2HNLWMRKF4ZAE6G5OXFTMK6ZBUU2ZD66ADA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.375190994 (111h26m38.926661455s) [noise=true , address=77.90.31.16:4734 , piece_id=O6W3SYGZO6O4JCAFDJNPP6UOFAFEN6DWSTKUI2ETOL4JZVTSWPLA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.421206747 (111h26m38.972677208s) [noise=true , address=45.91.116.7:4011 , piece_id=7P7OXLURT2X2Y4GPCKOKJHYRMCJNYOLASP6CXAOHHVTSLJG5XSKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.444750057 (111h26m38.996220509s) [noise=true , address=77.90.31.11:4554 , piece_id=C43IJLTGENL3XI7SJ6GLVP7ZKT5WXCEA4SB6QHJYWRPQCL3TCCSQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:30:40.446611513 (111h26m38.998081994s) [noise=true , address=45.91.116.6:4973 , piece_id=2F5OCJWVHDEHBTGUGEDIIELWZN22QFXXYEOTGLJDEKFKLPHOR5XA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:02.045460986 (111h27m0.596931447s) [noise=true , address=145.79.227.31:4797 , piece_id=EDBFHSFDRZMGZL7BXJGK3JUDTW4ZRQ5NONX5RHXGFHQJ3BKVZSJA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:05.177007325 (111h27m3.728477786s) [noise=true , address=145.79.227.18:4007 , piece_id=QKS5PKIEFELFAAVUHQWRC5G7BMDIWEIFJBO7BQ6RE2HQZVWAF3BA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:06.849449429 (111h27m5.40091989s) [noise=true , address=145.79.227.10:4402 , piece_id=Y2A4VQPAOJPIN6FDKHHG2UR6N2U2C5JG3PJ6PDD5MKTK7QD5MDUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: <nil> Nov 13 23:31:07.271630024 (111h27m5.823100485s) [noise=true , address=145.79.227.11:4731 , piece_id=YNZNJYAUKGRQBWCV33L2ZXYZNBVW3J3U66SIWJH5DZFALTRJOOUQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:145.79.227.11:4731): protocol: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51118->145.79.227.11:4731: use of closed network connection Nov 13 23:31:07.271895419 (111h27m5.82336587s) [noise=true , address=145.79.227.16:4910 , piece_id=ZBPK2AV4US4OLACTJKFE3DO5LJLBL4T5SWJHWGPKHTMXEJTNUDZQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35488->145.79.227.16:4910: use of closed network connection Nov 13 23:31:07.272020558 (111h27m5.823491019s) [noise=true , address=145.79.227.30:4761 , piece_id=3TF3IEJRQOQSTBDVDR622B27RZ5OTKZACJMKZ2JFAJAFHLE2YJAQ, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE, address:145.79.227.30:4761): protocol: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48116->145.79.227.30:4761: use of closed network connection Nov 13 23:31:07.272234275 (111h27m5.823704726s) [noise=true , address=145.79.227.21:4473 , piece_id=SOA7JZX3OZVHG7XEHGEXBPPWWQRZACDQSMHK7CVTKPQTEYS3ZQRA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:145.79.227.21:4473): protocol: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45942->145.79.227.21:4473: use of closed network connection Nov 13 23:31:07.272323725 (111h27m5.823794186s) [noise=true , address=145.79.227.22:4510 , piece_id=ESRB5JVFYJZ5TDLIA6BVDYOIZTK44TPQJHCUCSTD2XJOAQRD4A7A, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:145.79.227.22:4510): protocol: read tcp 188.166.225.170:38162->145.79.227.22:4510: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272452179 (111h27m5.82392264s) [noise=true , address=145.79.227.14:4839 , piece_id=ZD6SJ7UD7446MOWIUFMGAVPYTLUPMCASXHTZUZ734IEZSS4JLQ3Q, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44720->145.79.227.14:4839: use of closed network connection Nov 13 23:31:07.272575515 (111h27m5.824045976s) [noise=true , address=145.79.227.24:4582 , piece_id=NEQ7TD7ZIXJZCFGTD2MGLJJR7XSWC3RZ4VNSUXBHKMRTGMCVNXKA, seg_pos={0 0}, upload=98046]: Done uploading piece. err: ecclient: upload failed (node:1YLhczXWKCwzCQxM97Q77HpiWNMe5E15Nv679FZo5uHw6MRXa, address:145.79.227.24:4582): protocol: read tcp 188.166.225.170:56460->145.79.227.24:4582: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 13 23:31:07.272614158 (111h27m5.824084629s) [seg_pos={0 0}, upload=98046]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Nov 13 23:31:07.517953513 (111h27m6.069423974s) [upload=98046]: Done uploading segments... Nov 13 23:31:07.517993258 (111h27m6.069463719s) [upload=98046]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.248811702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.843821605s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:30:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.843856402s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........XX............X....X........X...X........XX........] duration: 6.07022915s trace: https://jaeger.dev.storj.io/trace/3933635ea304ab04 FAIL (2025-11-13 23:29:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 13 23:23:32.222110443 (111h19m30.773580904s) [upload=97944]: Starting upload Nov 13 23:23:32.222193351 (111h19m30.773663812s) [upload=97944]: Done starting upload Nov 13 23:23:32.222434801 (111h19m30.773905252s) [upload=97944]: Uploading segments... Nov 13 23:23:32.222535482 (111h19m30.774005943s) [upload=97944]: Got next segment. Inline: false Nov 13 23:23:32.222570429 (111h19m30.77404089s) [upload=97944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 13 23:23:32.222575719 (111h19m30.77404617s) [upload=97944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 13 23:23:32.489214273 (111h19m31.040684734s) [seg_pos={0 0}, upload=97944]: Begin upload segment... Nov 13 23:23:32.489262244 (111h19m31.040732695s) [seg_pos={0 0}, upload=97944]: Done begin upload segment. Nov 13 23:23:32.489288524 (111h19m31.040758975s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.489365290 (111h19m31.040835751s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490176184 (111h19m31.041646635s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490619007 (111h19m31.042089459s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.490959626 (111h19m31.042430077s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491307619 (111h19m31.04277808s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.491679226 (111h19m31.043149677s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.492333031 (111h19m31.043803492s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493445790 (111h19m31.044916251s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.493939951 (111h19m31.045410412s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495108186 (111h19m31.046578647s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.495243934 (111h19m31.046714395s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.496459299 (111h19m31.04792976s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.497730238 (111h19m31.049200689s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.498902531 (111h19m31.050372982s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:32.500159745 (111h19m31.051630206s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Uploading piece... Nov 13 23:23:34.168943842 (111h19m32.720414303s) [upload=97944]: Next returned nil segment Nov 13 23:23:34.168968589 (111h19m32.72043905s) [upload=97944]: Waiting for error group managing segments... Nov 13 23:23:34.566926491 (111h19m33.118396952s) [noise=true , address=77.90.31.18:4807 , piece_id=47H5SX3PJ7WXXHUD4D3LANVTB6QS3Y44LPTM3XLPMCYC6JJVMM6A, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.567727145 (111h19m33.119197596s) [noise=true , address=45.91.116.7:4010 , piece_id=7KUQZZANVYFAI4SJBFR5R3N6I27QMIX7GCISNX446TAGYVYC3CMA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.584836114 (111h19m33.136306575s) [noise=true , address=45.91.116.11:4117 , piece_id=FSLFZR3RZ44DHHDM4XUBZGTMJUTOSHEZOUOPBXH6IKN4HODMWPBQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.587038137 (111h19m33.138508598s) [noise=true , address=45.91.116.4:4937 , piece_id=XTKU7RGF4QCJCNIKEGPL7BTQNFKMR7IS3UXPGL464KXRZGHNWUDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.603143174 (111h19m33.154613625s) [noise=true , address=45.91.116.11:4118 , piece_id=I3IAYRA35U7RMGYOKDXK35BBLE4PGHO66WFP2TNIQDTIJAX6R6UA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.633392504 (111h19m33.184862955s) [noise=true , address=77.90.31.13:4627 , piece_id=SYXCBYZM2ONJSM3OOQEJ73GW2IEIZMJ2W5QBVZS7S2EPVTTUSPEQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.646566731 (111h19m33.198037182s) [noise=true , address=45.91.116.12:4227 , piece_id=RHVW5D2YJZXC27UBY5HXGHXOJUODHJFYH6TZTMLPQ53ATZ6FJTIQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.655804502 (111h19m33.207274963s) [noise=true , address=45.91.116.9:4263 , piece_id=SMP4LF3EBTXANDMGCJYAMROIM5QJMMHDTEMASQNYUIKCPOOUBSFQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:34.678400780 (111h19m33.229871231s) [noise=true , address=45.91.116.11:4119 , piece_id=QJTHTZWOHGZXCBVV3NIHDFV2DHXHW4IIJSHROUJBE43L3ZEVMMOA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:23:55.588855093 (111h19m54.140325554s) [noise=true , address=145.79.227.30:4762 , piece_id=CX2L6EFB7IDQF4JXQG267JIKYSR7IHC6TVA524YIFF3TG6UTUG5Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.058280180 (111h19m59.609750631s) [noise=true , address=145.79.227.8:4329 , piece_id=HXCUSSVTR2RNG7VOVJWVCGVAURI63IGLISTEDPOJTTJBUF44U5SA, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.487675183 (111h20m0.039145654s) [noise=true , address=145.79.227.9:4367 , piece_id=VDTDY4Y2MSUS23PH67CRFVQZOS4766H5VFAUCBDBWFVP5BPC5DDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: <nil> Nov 13 23:24:01.488290434 (111h20m0.039760905s) [noise=true , address=145.79.227.10:4402 , piece_id=YKYRSDKISM66B2XYCQONOJU23QQEEM2W3CMT7XTESVPDKOFGCXJQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12CMKNUtKvidnZpTfnYPxDCPUdvaSUs5mAx9oFFd2bWfuocTT4u, address:145.79.227.10:4402): protocol: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50576->145.79.227.10:4402: use of closed network connection Nov 13 23:24:01.488833839 (111h20m0.0403043s) [noise=true , address=145.79.227.16:4909 , piece_id=5EVVSLYQBBADKJYZ5APVJVMQ75O2FOMDAPSPLCSQKB4JCE4DUY6Q, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:145.79.227.16:4909): protocol: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42300->145.79.227.16:4909: use of closed network connection Nov 13 23:24:01.489149691 (111h20m0.040620152s) [noise=true , address=145.79.227.13:4802 , piece_id=D3H4OEJARRNAL57K5S6POE7QPQPECTF5MF2K4NS3MKXAPYJ2M3TQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:145.79.227.13:4802): protocol: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47014->145.79.227.13:4802: use of closed network connection Nov 13 23:24:01.489463719 (111h20m0.04093418s) [noise=true , address=145.79.227.20:4438 , piece_id=PTPFVPL3TE6LHPBFBWV62BVMKJJ6UASD5MFDIYXBORR3WWVCUXDQ, seg_pos={0 0}, upload=97944]: Done uploading piece. err: ecclient: upload failed (node:1GEBfCYivHXYjSXHLmJjrGbgXh4UATzZBDzvx7oLTzeTHum8PT, address:145.79.227.20:4438): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:01.489493796 (111h20m0.040964247s) [seg_pos={0 0}, upload=97944]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 13 23:24:01.748156531 (111h20m0.299627002s) [upload=97944]: Done uploading segments... Nov 13 23:24:01.748206497 (111h20m0.299676958s) [upload=97944]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.070262965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.68890637s trace: https://jaeger.dev.storj.io/trace/5314e7986aeb2e66 FAIL (2025-11-13 23:29:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 111h19m2.830044835s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279158486 (111h19m2.830628947s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.279532047 (111h19m2.831002508s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280300521 (111h19m2.831770982s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280373660 (111h19m2.831844121s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280765076 (111h19m2.832235537s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.280777018 (111h19m2.832247479s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281155559 (111h19m2.83262602s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281191307 (111h19m2.832661768s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281536074 (111h19m2.833006535s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281629872 (111h19m2.833100333s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.281925565 (111h19m2.833396026s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282082964 (111h19m2.833553426s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282523053 (111h19m2.833993514s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.282967399 (111h19m2.83443786s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284119753 (111h19m2.835590224s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:04.284617892 (111h19m2.836088363s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Uploading piece... Nov 13 23:23:05.654655793 (111h19m4.206126254s) [upload=97934]: Next returned nil segment Nov 13 23:23:05.654669690 (111h19m4.206140142s) [upload=97934]: Waiting for error group managing segments... Nov 13 23:23:06.020714492 (111h19m4.572184943s) [noise=true , address=64.49.13.137:4503 , piece_id=674WWWPCVVQ3G74IR53RB2ZSSXP3EYMFLNQZNUGWONC6DJPYHSEA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.042306798 (111h19m4.593777249s) [noise=true , address=5.231.73.2:4017 , piece_id=3FZ4YJUFRQ55KB45ZKAXXNXKRFIIJCEU2I6MQZYSA56UEKZDQUTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.063229168 (111h19m4.614699619s) [noise=true , address=5.231.73.4:4014 , piece_id=2X7CRRH4BBUF3HTJS2GYEGH4ABTX6ZN2RIW6BYXBS7HANQXF2K5A, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.123581201 (111h19m4.675051662s) [noise=true , address=5.231.73.5:4004 , piece_id=4EJHANO6M5LTN4SB62UVITDE6L6HHLQ4FNE54NZMMIDYMTPUIYCA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.249919163 (111h19m4.801389624s) [noise=true , address=45.91.116.13:4158 , piece_id=S4SBQG33YRT2PH6YHOEFGZBEUWU7K6DJ3BCUH4NI577KZENRBGXA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.250341287 (111h19m4.801811758s) [noise=true , address=45.91.116.5:4323 , piece_id=WVYEZAV3MAN44GXCN2SMHKAT6UWP64FBG4V5D5EKCD6QSWIONGYQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.273701780 (111h19m4.825172231s) [noise=true , address=45.91.116.12:4235 , piece_id=3N3DPVWKLQRGRYW4LJLPW6J4DIEQNY76YWAFKGP7Y55RDBAP7TGA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.276342829 (111h19m4.82781328s) [noise=true , address=77.90.31.19:4874 , piece_id=7YQRZ234EV22G3VWJVDJSNHDR3WPHRS42DJTDVMMMQFDHWD65STA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.286362338 (111h19m4.837832799s) [noise=true , address=77.90.31.17:4773 , piece_id=4OG4AIQKFKDWSOGOHGEZQQNZLKMID46TFY2DCZGNC6KT5RDO5W3Q, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288026447 (111h19m4.839496908s) [noise=true , address=77.90.31.18:4838 , piece_id=SWUIEG6INLEVSJS4LLRI4TWT3PXS2FOG4BPDNUOJIHSDB6EX7QWQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.288632520 (111h19m4.840102981s) [noise=true , address=77.90.31.12:4613 , piece_id=UTMWJBMRLPOZD6NXZFGWVCHP57LEI5YM7TXUK4STZ2VNB6O5EQZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:23:06.310706564 (111h19m4.862177025s) [noise=true , address=77.90.31.13:4652 , piece_id=2H4W7VC7AOUH6H655TBY7J7CYE7VVVOI6IUAFE6Z3GGDVFFUWJLA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: <nil> Nov 13 23:24:03.313647487 (111h20m1.865117958s) [noise=true , address=145.79.227.34:4912 , piece_id=OSN5SLFMOU2AXPZSES344EADFT5R36AEGBPNBCGXMW3FMBXO7WDA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12HnqBnErwajHUpuLrYCmCg7cbJYMQohVM2pQbHZ9LFCvjye39x, address:145.79.227.34:4912): protocol: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59246->145.79.227.34:4912: use of closed network connection Nov 13 23:24:03.314209678 (111h20m1.865680149s) [noise=true , address=145.79.227.45:4234 , piece_id=MEUGEELK7KLNNVHOFBIYMCUCT7VCGAQQ4VZBBOCL3YHNV5ZCXSQA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12uthP31WaBmn37h54Q6z9ZMtYRY6K8917Ms4kwyyADqu4EXPJy, address:145.79.227.45:4234): protocol: expected piece hash; context deadline exceeded; EOF Nov 13 23:24:03.314476546 (111h20m1.865947017s) [noise=true , address=145.79.227.48:4306 , piece_id=VKFTILEEA6KJL3LKST6FGP7W5YGGDIWHCSQK4ZDCKV65OELLQJZA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12amBPo7WKpjPDzi6mUWw5PcdgrcT7hsUjsSkijfrvKka1bJzNw, address:145.79.227.48:4306): protocol: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45508->145.79.227.48:4306: use of closed network connection Nov 13 23:24:03.314681156 (111h20m1.866151617s) [noise=true , address=145.79.227.16:4937 , piece_id=G6QGV57MJ4SJKEM7UUSOVVF4UBCCQVKDWCESQHAXJX2GFQCD4PIQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:1244drkeyc94qojgXAjt8oenBaGpzQBiMmisCjfBys5sLW4GvqU, address:145.79.227.16:4937): protocol: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33708->145.79.227.16:4937: use of closed network connection Nov 13 23:24:03.314822956 (111h20m1.866293427s) [noise=true , address=145.79.227.37:4573 , piece_id=AM2JBLBRL5FCVD5DQBBBW5WH26RDH3NUC7S5VJUHRGLAZ5LIF7XA, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:12ja5PugYok1rEPimj7ijuesn2JGouDztPg3GwUnzseAywgkfnx, address:145.79.227.37:4573): protocol: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42174->145.79.227.37:4573: use of closed network connection Nov 13 23:24:03.315331764 (111h20m1.866802235s) [noise=true , address=145.79.227.41:4118 , piece_id=TBZHKJI6LZWFSVQOF3NMDAXLKVVTXWEGPX5O5A4FC4YBUT2ZRTTQ, seg_pos={0 0}, upload=97934]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:145.79.227.41:4118): protocol: read tcp 188.166.225.170:43354->145.79.227.41:4118: use of closed network connection; context deadline exceeded Nov 13 23:24:03.315377802 (111h20m1.866848264s) [seg_pos={0 0}, upload=97934]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Nov 13 23:24:03.564632753 (111h20m2.116103244s) [upload=97934]: Done uploading segments... Nov 13 23:24:03.564681767 (111h20m2.116152228s) [upload=97934]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 5.688934174s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)