Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 38.123049831s FAIL (2025-09-27 11:51:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.123080671s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [..................................................X.X..XX...] duration: 6.177582832s FAIL (2025-09-27 11:50:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7:36.492060325 (94h16m21.954152808s) [upload=82949]: Uploading segments... Sep 27 11:47:36.492195170 (94h16m21.954287654s) [upload=82949]: Got next segment. Inline: false Sep 27 11:47:36.492234677 (94h16m21.95432715s) [upload=82949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:47:36.492248272 (94h16m21.954340755s) [upload=82949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:47:36.757237199 (94h16m22.219329683s) [seg_pos={0 0}, upload=82949]: Begin upload segment... Sep 27 11:47:36.757285672 (94h16m22.219378155s) [seg_pos={0 0}, upload=82949]: Done begin upload segment. Sep 27 11:47:36.757310539 (94h16m22.219403012s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.758667182 (94h16m22.220759655s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.759559131 (94h16m22.221651614s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.760219388 (94h16m22.222311861s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.764491273 (94h16m22.226583756s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.765278912 (94h16m22.227371385s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.766412391 (94h16m22.228504874s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.767804031 (94h16m22.229896504s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.769163159 (94h16m22.231255632s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.770397880 (94h16m22.232490363s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.771590040 (94h16m22.233682513s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.772001854 (94h16m22.234094337s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774122532 (94h16m22.236215015s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774784743 (94h16m22.236877226s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.775821457 (94h16m22.23791394s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.780394705 (94h16m22.242487188s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:38.461586904 (94h16m23.923679377s) [upload=82949]: Next returned nil segment Sep 27 11:47:38.461610158 (94h16m23.923702631s) [upload=82949]: Waiting for error group managing segments... Sep 27 11:47:38.740459128 (94h16m24.202551601s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.745859060 (94h16m24.207951543s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.747709353 (94h16m24.209801826s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.756425361 (94h16m24.218517844s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.763830400 (94h16m24.225922883s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.034790168 (94h16m24.496882641s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.082992984 (94h16m24.545085468s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.093503058 (94h16m24.555595531s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:48:05.776650727 (94h16m51.23874322s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 188.166.225.170:41684->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776791365 (94h16m51.238883848s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776940920 (94h16m51.239033403s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 188.166.225.170:39958->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776961940 (94h16m51.239054423s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:46556->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777060468 (94h16m51.239152981s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777103098 (94h16m51.239195582s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:36106->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777161320 (94h16m51.239253793s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49532->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777250079 (94h16m51.239342562s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:47228->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:48:05.777277370 (94h16m51.239369843s) [seg_pos={0 0}, upload=82949]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 27 11:48:06.040681493 (94h16m51.502773977s) [upload=82949]: Done uploading segments... Sep 27 11:48:06.040717993 (94h16m51.502810476s) [upload=82949]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.177700315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X..................................................X.X..XX..] duration: 5.920019741s FAIL (2025-09-27 11:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7:36.492060325 (94h16m21.954152808s) [upload=82949]: Uploading segments... Sep 27 11:47:36.492195170 (94h16m21.954287654s) [upload=82949]: Got next segment. Inline: false Sep 27 11:47:36.492234677 (94h16m21.95432715s) [upload=82949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:47:36.492248272 (94h16m21.954340755s) [upload=82949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:47:36.757237199 (94h16m22.219329683s) [seg_pos={0 0}, upload=82949]: Begin upload segment... Sep 27 11:47:36.757285672 (94h16m22.219378155s) [seg_pos={0 0}, upload=82949]: Done begin upload segment. Sep 27 11:47:36.757310539 (94h16m22.219403012s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.758667182 (94h16m22.220759655s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.759559131 (94h16m22.221651614s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.760219388 (94h16m22.222311861s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.764491273 (94h16m22.226583756s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.765278912 (94h16m22.227371385s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.766412391 (94h16m22.228504874s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.767804031 (94h16m22.229896504s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.769163159 (94h16m22.231255632s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.770397880 (94h16m22.232490363s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.771590040 (94h16m22.233682513s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.772001854 (94h16m22.234094337s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774122532 (94h16m22.236215015s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774784743 (94h16m22.236877226s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.775821457 (94h16m22.23791394s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.780394705 (94h16m22.242487188s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:38.461586904 (94h16m23.923679377s) [upload=82949]: Next returned nil segment Sep 27 11:47:38.461610158 (94h16m23.923702631s) [upload=82949]: Waiting for error group managing segments... Sep 27 11:47:38.740459128 (94h16m24.202551601s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.745859060 (94h16m24.207951543s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.747709353 (94h16m24.209801826s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.756425361 (94h16m24.218517844s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.763830400 (94h16m24.225922883s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.034790168 (94h16m24.496882641s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.082992984 (94h16m24.545085468s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.093503058 (94h16m24.555595531s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:48:05.776650727 (94h16m51.23874322s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 188.166.225.170:41684->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776791365 (94h16m51.238883848s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776940920 (94h16m51.239033403s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 188.166.225.170:39958->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776961940 (94h16m51.239054423s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:46556->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777060468 (94h16m51.239152981s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777103098 (94h16m51.239195582s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:36106->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777161320 (94h16m51.239253793s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49532->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777250079 (94h16m51.239342562s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:47228->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:48:05.777277370 (94h16m51.239369843s) [seg_pos={0 0}, upload=82949]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 27 11:48:06.040681493 (94h16m51.502773977s) [upload=82949]: Done uploading segments... Sep 27 11:48:06.040717993 (94h16m51.502810476s) [upload=82949]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.920112878s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 14.093902222s FAIL (2025-09-27 11:49:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 14.093950474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.249752004s FAIL (2025-09-27 11:49:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.249787512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X..................................................X.X..XX.] duration: 7.678183461s FAIL (2025-09-27 11:48:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7:36.492060325 (94h16m21.954152808s) [upload=82949]: Uploading segments... Sep 27 11:47:36.492195170 (94h16m21.954287654s) [upload=82949]: Got next segment. Inline: false Sep 27 11:47:36.492234677 (94h16m21.95432715s) [upload=82949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:47:36.492248272 (94h16m21.954340755s) [upload=82949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:47:36.757237199 (94h16m22.219329683s) [seg_pos={0 0}, upload=82949]: Begin upload segment... Sep 27 11:47:36.757285672 (94h16m22.219378155s) [seg_pos={0 0}, upload=82949]: Done begin upload segment. Sep 27 11:47:36.757310539 (94h16m22.219403012s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.758667182 (94h16m22.220759655s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.759559131 (94h16m22.221651614s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.760219388 (94h16m22.222311861s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.764491273 (94h16m22.226583756s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.765278912 (94h16m22.227371385s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.766412391 (94h16m22.228504874s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.767804031 (94h16m22.229896504s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.769163159 (94h16m22.231255632s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.770397880 (94h16m22.232490363s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.771590040 (94h16m22.233682513s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.772001854 (94h16m22.234094337s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774122532 (94h16m22.236215015s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774784743 (94h16m22.236877226s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.775821457 (94h16m22.23791394s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.780394705 (94h16m22.242487188s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:38.461586904 (94h16m23.923679377s) [upload=82949]: Next returned nil segment Sep 27 11:47:38.461610158 (94h16m23.923702631s) [upload=82949]: Waiting for error group managing segments... Sep 27 11:47:38.740459128 (94h16m24.202551601s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.745859060 (94h16m24.207951543s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.747709353 (94h16m24.209801826s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.756425361 (94h16m24.218517844s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.763830400 (94h16m24.225922883s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.034790168 (94h16m24.496882641s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.082992984 (94h16m24.545085468s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.093503058 (94h16m24.555595531s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:48:05.776650727 (94h16m51.23874322s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 188.166.225.170:41684->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776791365 (94h16m51.238883848s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776940920 (94h16m51.239033403s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 188.166.225.170:39958->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776961940 (94h16m51.239054423s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:46556->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777060468 (94h16m51.239152981s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777103098 (94h16m51.239195582s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:36106->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777161320 (94h16m51.239253793s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49532->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777250079 (94h16m51.239342562s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:47228->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:48:05.777277370 (94h16m51.239369843s) [seg_pos={0 0}, upload=82949]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 27 11:48:06.040681493 (94h16m51.502773977s) [upload=82949]: Done uploading segments... Sep 27 11:48:06.040717993 (94h16m51.502810476s) [upload=82949]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 7.678216785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 13.614925956s FAIL (2025-09-27 11:48:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 13.61495925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X..................................................X.X..XX] duration: 30.266355466s FAIL (2025-09-27 11:48:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 7:36.492060325 (94h16m21.954152808s) [upload=82949]: Uploading segments... Sep 27 11:47:36.492195170 (94h16m21.954287654s) [upload=82949]: Got next segment. Inline: false Sep 27 11:47:36.492234677 (94h16m21.95432715s) [upload=82949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:47:36.492248272 (94h16m21.954340755s) [upload=82949]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:47:36.757237199 (94h16m22.219329683s) [seg_pos={0 0}, upload=82949]: Begin upload segment... Sep 27 11:47:36.757285672 (94h16m22.219378155s) [seg_pos={0 0}, upload=82949]: Done begin upload segment. Sep 27 11:47:36.757310539 (94h16m22.219403012s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.758667182 (94h16m22.220759655s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.759559131 (94h16m22.221651614s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.760219388 (94h16m22.222311861s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.764491273 (94h16m22.226583756s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.765278912 (94h16m22.227371385s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.766412391 (94h16m22.228504874s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.767804031 (94h16m22.229896504s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.769163159 (94h16m22.231255632s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.770397880 (94h16m22.232490363s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.771590040 (94h16m22.233682513s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.772001854 (94h16m22.234094337s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774122532 (94h16m22.236215015s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.774784743 (94h16m22.236877226s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.775821457 (94h16m22.23791394s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:36.780394705 (94h16m22.242487188s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Uploading piece... Sep 27 11:47:38.461586904 (94h16m23.923679377s) [upload=82949]: Next returned nil segment Sep 27 11:47:38.461610158 (94h16m23.923702631s) [upload=82949]: Waiting for error group managing segments... Sep 27 11:47:38.740459128 (94h16m24.202551601s) [noise=true , address=145.79.227.16:4911 , piece_id=27BC6BECOTFZGXG3RVYGWYR3IKFV3YHUEQOUS3KZJUG664XCZQKA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.745859060 (94h16m24.207951543s) [noise=true , address=145.79.227.11:4730 , piece_id=6TESVTPPB5RGISFILBFL3A3EH4O5H5VPZZIP66AJMPZQORU7YEBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.747709353 (94h16m24.209801826s) [noise=true , address=145.79.227.17:4969 , piece_id=U24QT6DFPKRYKX7BI57D3YH4VGRMD36TZ7TPGG7X2WKFPFFJNXSA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.756425361 (94h16m24.218517844s) [noise=true , address=145.79.227.4:4187 , piece_id=MGENYKOC7KMRZ7ZHVJB7YEWSLHNMTKKDHVIT6LC6MEN6QDRI6RAA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:38.763830400 (94h16m24.225922883s) [noise=true , address=145.79.227.6:4257 , piece_id=LWP6T4FACINGYXJFU2OQA5MZBP6ZZ7KPURKJ6MZJ2IHHBYVCLMJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.034790168 (94h16m24.496882641s) [noise=true , address=77.90.31.15:4697 , piece_id=REM77UCIX2H3F3ZY67BJUI7NDA6Z5RLKWCAZJJOQN4H6AOK73EJQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.082992984 (94h16m24.545085468s) [noise=true , address=77.90.31.14:4662 , piece_id=TWO7TWCSKLFTOGUNK25Y56EALEZLB6IECVZ25HCJFI5XXCXMGYEQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:47:39.093503058 (94h16m24.555595531s) [noise=true , address=77.90.31.16:4734 , piece_id=UO55NBO72Z5AUKVLHBAMWM5OIA2FNRIBHF6A5BAMF2LR2WVP6RBQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: <nil> Sep 27 11:48:05.776650727 (94h16m51.23874322s) [noise=true , address=45.91.116.6:4975 , piece_id=QO3XXUGFB4BABPO36R7UDZNCFL4CO26B6ODI6IKSH3UQRGANRF7A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, address:45.91.116.6:4975): protocol: write tcp 188.166.225.170:41684->45.91.116.6:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776791365 (94h16m51.238883848s) [noise=true , address=45.91.116.9:4262 , piece_id=QPZY45Z75ZEVIMKY5RFI3O6EQXPIRF2MS53CUQ5BX4RILOMP6JJA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: rpc: dial tcp 45.91.116.9:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776940920 (94h16m51.239033403s) [noise=true , address=45.91.116.9:4263 , piece_id=MOVTQ2T7GNJZQYJDDXW4SQMWYJE45OU5O7GZBQKQWN3O7OCGYY6A, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, address:45.91.116.9:4263): protocol: write tcp 188.166.225.170:39958->45.91.116.9:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.776961940 (94h16m51.239054423s) [noise=true , address=45.91.116.13:4153 , piece_id=FHWUNRTYQSCQA3Z3SVWEABJQVHTV2U72FQLGFHEBF3OHVNB7VXHQ, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: write tcp 188.166.225.170:46556->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777060468 (94h16m51.239152981s) [noise=true , address=45.91.116.8:4047 , piece_id=PEPXGOLY2ZF56R66HJC22CR3HMDK3ON4VS7LNM5OLW4E6GWUCLVA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777103098 (94h16m51.239195582s) [noise=true , address=45.91.116.5:4299 , piece_id=ZMGXGTZVJBQAEMXSHHSFKAPKSWQNVVN6253UDVDZ65T3KG3JI5EA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:36106->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777161320 (94h16m51.239253793s) [noise=true , address=45.91.116.14:4190 , piece_id=2EDZEKNKR6EW4FFTHSZVCYB6M4FNJCVFTD5I3WNX6Q4LFJAHG6AA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: write tcp 188.166.225.170:49532->45.91.116.14:4190: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:48:05.777250079 (94h16m51.239342562s) [noise=true , address=45.91.116.4:4940 , piece_id=4CUDOSYBOTMC5OB5OFJ75YKKNYEMTEHKGSG2FKJZIKZNXWEDI2CA, seg_pos={0 0}, upload=82949]: Done uploading piece. err: ecclient: upload failed (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe, address:45.91.116.4:4940): protocol: write tcp 188.166.225.170:47228->45.91.116.4:4940: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:48:05.777277370 (94h16m51.239369843s) [seg_pos={0 0}, upload=82949]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 27 11:48:06.040681493 (94h16m51.502773977s) [upload=82949]: Done uploading segments... Sep 27 11:48:06.040717993 (94h16m51.502810476s) [upload=82949]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.266372278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..................................................X.X..X] duration: 30.266286374s FAIL (2025-09-27 11:47:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== Sep 27 11:47:06.004398400 (94h15m51.466490883s) [upload=82942]: Starting upload Sep 27 11:47:06.004538216 (94h15m51.466630699s) [upload=82942]: Done starting upload Sep 27 11:47:06.004738277 (94h15m51.46683076s) [upload=82942]: Uploading segments... Sep 27 11:47:06.004905145 (94h15m51.466997628s) [upload=82942]: Got next segment. Inline: false Sep 27 11:47:06.004968845 (94h15m51.467061329s) [upload=82942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:47:06.004975629 (94h15m51.467068112s) [upload=82942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:47:06.264945398 (94h15m51.727037881s) [seg_pos={0 0}, upload=82942]: Begin upload segment... Sep 27 11:47:06.265204471 (94h15m51.727296954s) [seg_pos={0 0}, upload=82942]: Done begin upload segment. Sep 27 11:47:06.265254387 (94h15m51.72734687s) [noise=true , address=77.90.31.19:4843 , piece_id=3OC77MVYJXMQIE6FWSSCAKPCRSUYOSZX74CZ4HHTSNUB3KGBX72A, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.265275767 (94h15m51.72736825s) [noise=true , address=145.79.227.2:4114 , piece_id=NTLZAXXGPFHDONH64SARWXT3YPZZSRAZZUTWDAMEIASXJP7SBHJA, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.266027669 (94h15m51.728120152s) [noise=true , address=45.91.116.11:4118 , piece_id=TY2YXP45YX7XNBDABVGTKBNVPZZACJHNYM4FT2TIPUT675CIL26A, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.266418263 (94h15m51.728510746s) [noise=true , address=145.79.227.3:4150 , piece_id=LKDWJM3IVQRTFVOSPTKAKFVNVIQEJFW7365ZFTPEGGYONG34WP7Q, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.266555855 (94h15m51.728648338s) [noise=true , address=77.90.31.11:4555 , piece_id=3CXSI2GSCGBYDPT4BGFTLN7TO7RBX2B7RNJVH73X2UK6AN5OFOTQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.267009849 (94h15m51.729102322s) [noise=true , address=45.91.116.5:4299 , piece_id=S7K5QF7B2QKODRI2AJSA4XNQAUW7NP4DPGQI4Q3KXA46HCXH5GGQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.267285514 (94h15m51.729377997s) [noise=true , address=45.91.116.6:4973 , piece_id=CQDFDPT6SLJNUEV76PUE2UWI5SJYNNLU4FTFFMSM5XLOK5CKXHTQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.267697268 (94h15m51.729789751s) [noise=true , address=45.91.116.12:4227 , piece_id=HOERIUW5W6N5VZPYMQZ6WMTVU6NHOACZMAJ4AHBWIDXSEQC2I7VQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.268098944 (94h15m51.730191427s) [noise=true , address=77.90.31.21:4878 , piece_id=3KAQ45ELOJMYB66TOOYWJXLG3GSRSAU7LKLJTIC2WSP4KXBNNEBA, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.268371953 (94h15m51.730464436s) [noise=true , address=145.79.227.16:4911 , piece_id=2HWINRFWPQ2BZOXAL5JZAZD4EHHY4OXWKXZ76B2254NXMDTBSEAQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.268897635 (94h15m51.730990118s) [noise=true , address=45.91.116.14:4191 , piece_id=IVEZ76ZVARXVDNFFNKAAHGPY5BVSHOYTCKXTN5I4KYOWPJAHRIZQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.268930026 (94h15m51.731022519s) [noise=true , address=77.90.31.17:4771 , piece_id=NP7UFYW3EFHZCG3UNY2JMRII24WZOYB3KXNYASX2EA6QYBFPI4PA, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.269375654 (94h15m51.731468137s) [noise=true , address=145.79.227.13:4801 , piece_id=S62YD2X2UOA36WMHAKGK2RIRIVU2XM5WB36B4GZ5CUJNO2FUSDZA, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.269761099 (94h15m51.731853582s) [noise=true , address=45.91.116.10:4082 , piece_id=GBKTSDKNCLKYUVJJG5ALTXOVJRDTFIXNEBJBR576QPRKF7MKZO5Q, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.270168204 (94h15m51.732260687s) [noise=true , address=77.90.31.10:4518 , piece_id=5XUWYPWNKSK6DSUO3FBVLUEWOJDZWZGLO44UVLUBBZ26GLMI2XFQ, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:06.270638209 (94h15m51.732730692s) [noise=true , address=45.91.116.8:4045 , piece_id=KS3Y7O4J4EKULP7V6QFA23M5HVCZ7TY4O5DIHT2TCYGLJNQXXHDA, seg_pos={0 0}, upload=82942]: Uploading piece... Sep 27 11:47:07.965479988 (94h15m53.427572461s) [upload=82942]: Next returned nil segment Sep 27 11:47:07.965497512 (94h15m53.427589995s) [upload=82942]: Waiting for error group managing segments... Sep 27 11:47:08.239847054 (94h15m53.701939537s) [noise=true , address=145.79.227.2:4114 , piece_id=NTLZAXXGPFHDONH64SARWXT3YPZZSRAZZUTWDAMEIASXJP7SBHJA, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:08.247169847 (94h15m53.70926232s) [noise=true , address=145.79.227.13:4801 , piece_id=S62YD2X2UOA36WMHAKGK2RIRIVU2XM5WB36B4GZ5CUJNO2FUSDZA, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:08.271951296 (94h15m53.734043779s) [noise=true , address=145.79.227.16:4911 , piece_id=2HWINRFWPQ2BZOXAL5JZAZD4EHHY4OXWKXZ76B2254NXMDTBSEAQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:10.105604549 (94h15m55.567697032s) [noise=true , address=145.79.227.3:4150 , piece_id=LKDWJM3IVQRTFVOSPTKAKFVNVIQEJFW7365ZFTPEGGYONG34WP7Q, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:12.056130377 (94h15m57.51822286s) [noise=true , address=77.90.31.19:4843 , piece_id=3OC77MVYJXMQIE6FWSSCAKPCRSUYOSZX74CZ4HHTSNUB3KGBX72A, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:12.149648492 (94h15m57.611740965s) [noise=true , address=77.90.31.21:4878 , piece_id=3KAQ45ELOJMYB66TOOYWJXLG3GSRSAU7LKLJTIC2WSP4KXBNNEBA, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:12.190855929 (94h15m57.652948412s) [noise=true , address=77.90.31.11:4555 , piece_id=3CXSI2GSCGBYDPT4BGFTLN7TO7RBX2B7RNJVH73X2UK6AN5OFOTQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:12.192266295 (94h15m57.654358778s) [noise=true , address=77.90.31.17:4771 , piece_id=NP7UFYW3EFHZCG3UNY2JMRII24WZOYB3KXNYASX2EA6QYBFPI4PA, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:12.361973388 (94h15m57.824065871s) [noise=true , address=77.90.31.10:4518 , piece_id=5XUWYPWNKSK6DSUO3FBVLUEWOJDZWZGLO44UVLUBBZ26GLMI2XFQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: <nil> Sep 27 11:47:35.260429506 (94h16m20.722521979s) [noise=true , address=45.91.116.8:4045 , piece_id=KS3Y7O4J4EKULP7V6QFA23M5HVCZ7TY4O5DIHT2TCYGLJNQXXHDA, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: write tcp 188.166.225.170:33244->45.91.116.8:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260517893 (94h16m20.722610367s) [noise=true , address=45.91.116.5:4299 , piece_id=S7K5QF7B2QKODRI2AJSA4XNQAUW7NP4DPGQI4Q3KXA46HCXH5GGQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, address:45.91.116.5:4299): protocol: write tcp 188.166.225.170:39332->45.91.116.5:4299: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260535958 (94h16m20.722628441s) [noise=true , address=45.91.116.11:4118 , piece_id=TY2YXP45YX7XNBDABVGTKBNVPZZACJHNYM4FT2TIPUT675CIL26A, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: rpc: dial tcp 45.91.116.11:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260650947 (94h16m20.72274342s) [noise=true , address=45.91.116.10:4082 , piece_id=GBKTSDKNCLKYUVJJG5ALTXOVJRDTFIXNEBJBR576QPRKF7MKZO5Q, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: rpc: dial tcp 45.91.116.10:4082: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260670966 (94h16m20.722763449s) [noise=true , address=45.91.116.6:4973 , piece_id=CQDFDPT6SLJNUEV76PUE2UWI5SJYNNLU4FTFFMSM5XLOK5CKXHTQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: rpc: dial tcp 45.91.116.6:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260688839 (94h16m20.722781312s) [noise=true , address=45.91.116.12:4227 , piece_id=HOERIUW5W6N5VZPYMQZ6WMTVU6NHOACZMAJ4AHBWIDXSEQC2I7VQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:47:35.260770996 (94h16m20.722863469s) [noise=true , address=45.91.116.14:4191 , piece_id=IVEZ76ZVARXVDNFFNKAAHGPY5BVSHOYTCKXTN5I4KYOWPJAHRIZQ, seg_pos={0 0}, upload=82942]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: write tcp 188.166.225.170:34016->45.91.116.14:4191: use of closed network connection; context deadline exceeded Sep 27 11:47:35.260797045 (94h16m20.722889528s) [seg_pos={0 0}, upload=82942]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Sep 27 11:47:35.525305917 (94h16m20.98739848s) [upload=82942]: Done uploading segments... Sep 27 11:47:35.525339842 (94h16m20.987432315s) [upload=82942]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.2663159s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 6.424085465s FAIL (2025-09-27 11:47:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.424116284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 8.644281348s FAIL (2025-09-27 11:46:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.644310322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.701990173s FAIL (2025-09-27 11:46:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.702023106s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X..................................................X.X..] duration: 6.745837322s FAIL (2025-09-27 11:46:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:45:21.574962251 (94h14m7.037054724s) [upload=82914]: Starting upload Sep 27 11:45:21.575054808 (94h14m7.037147291s) [upload=82914]: Done starting upload Sep 27 11:45:21.575156011 (94h14m7.037248534s) [upload=82914]: Uploading segments... Sep 27 11:45:21.575298221 (94h14m7.037390704s) [upload=82914]: Got next segment. Inline: false Sep 27 11:45:21.575351092 (94h14m7.037443576s) [upload=82914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:45:21.575356974 (94h14m7.037449447s) [upload=82914]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:45:21.835960057 (94h14m7.298052541s) [seg_pos={0 0}, upload=82914]: Begin upload segment... Sep 27 11:45:21.836030091 (94h14m7.298122564s) [seg_pos={0 0}, upload=82914]: Done begin upload segment. Sep 27 11:45:21.836060489 (94h14m7.298152972s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836070258 (94h14m7.298162731s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836583224 (94h14m7.298675707s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837000709 (94h14m7.299093192s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837468460 (94h14m7.299560933s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.839078847 (94h14m7.30117133s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840285635 (94h14m7.302378108s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840667893 (94h14m7.302760376s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.842333605 (94h14m7.304426098s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.844189488 (94h14m7.306281971s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.845791328 (94h14m7.307883812s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.846640675 (94h14m7.308733158s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.847437573 (94h14m7.309530056s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848493895 (94h14m7.310586378s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848964320 (94h14m7.311056793s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.850072000 (94h14m7.312164483s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:23.531091188 (94h14m8.993183671s) [upload=82914]: Next returned nil segment Sep 27 11:45:23.531113992 (94h14m8.993206475s) [upload=82914]: Waiting for error group managing segments... Sep 27 11:45:23.811944975 (94h14m9.274037458s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.820431265 (94h14m9.282523748s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.824197606 (94h14m9.286290089s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.838835901 (94h14m9.300928374s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845110639 (94h14m9.307203122s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845609919 (94h14m9.307702402s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.848133685 (94h14m9.310226168s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.964416249 (94h14m9.426508732s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.966287362 (94h14m9.428379845s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.970573033 (94h14m9.432665506s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.979664915 (94h14m9.441757398s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:50.846870042 (94h14m36.308962525s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.846936599 (94h14m36.309029102s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:37436->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847116211 (94h14m36.309208694s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:59280->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847189230 (94h14m36.309281713s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:47474->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847343314 (94h14m36.309435797s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:40326->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:45:50.847384322 (94h14m36.309476795s) [seg_pos={0 0}, upload=82914]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:45:51.096210372 (94h14m36.558302856s) [upload=82914]: Done uploading segments... Sep 27 11:45:51.096247213 (94h14m36.558339696s) [upload=82914]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 6.745868982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.113137623s FAIL (2025-09-27 11:46:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.113169564s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X....X..................................................X.X.] duration: 7.606123977s FAIL (2025-09-27 11:45:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:45:21.574962251 (94h14m7.037054724s) [upload=82914]: Starting upload Sep 27 11:45:21.575054808 (94h14m7.037147291s) [upload=82914]: Done starting upload Sep 27 11:45:21.575156011 (94h14m7.037248534s) [upload=82914]: Uploading segments... Sep 27 11:45:21.575298221 (94h14m7.037390704s) [upload=82914]: Got next segment. Inline: false Sep 27 11:45:21.575351092 (94h14m7.037443576s) [upload=82914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:45:21.575356974 (94h14m7.037449447s) [upload=82914]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:45:21.835960057 (94h14m7.298052541s) [seg_pos={0 0}, upload=82914]: Begin upload segment... Sep 27 11:45:21.836030091 (94h14m7.298122564s) [seg_pos={0 0}, upload=82914]: Done begin upload segment. Sep 27 11:45:21.836060489 (94h14m7.298152972s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836070258 (94h14m7.298162731s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836583224 (94h14m7.298675707s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837000709 (94h14m7.299093192s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837468460 (94h14m7.299560933s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.839078847 (94h14m7.30117133s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840285635 (94h14m7.302378108s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840667893 (94h14m7.302760376s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.842333605 (94h14m7.304426098s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.844189488 (94h14m7.306281971s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.845791328 (94h14m7.307883812s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.846640675 (94h14m7.308733158s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.847437573 (94h14m7.309530056s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848493895 (94h14m7.310586378s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848964320 (94h14m7.311056793s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.850072000 (94h14m7.312164483s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:23.531091188 (94h14m8.993183671s) [upload=82914]: Next returned nil segment Sep 27 11:45:23.531113992 (94h14m8.993206475s) [upload=82914]: Waiting for error group managing segments... Sep 27 11:45:23.811944975 (94h14m9.274037458s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.820431265 (94h14m9.282523748s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.824197606 (94h14m9.286290089s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.838835901 (94h14m9.300928374s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845110639 (94h14m9.307203122s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845609919 (94h14m9.307702402s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.848133685 (94h14m9.310226168s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.964416249 (94h14m9.426508732s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.966287362 (94h14m9.428379845s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.970573033 (94h14m9.432665506s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.979664915 (94h14m9.441757398s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:50.846870042 (94h14m36.308962525s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.846936599 (94h14m36.309029102s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:37436->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847116211 (94h14m36.309208694s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:59280->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847189230 (94h14m36.309281713s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:47474->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847343314 (94h14m36.309435797s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:40326->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:45:50.847384322 (94h14m36.309476795s) [seg_pos={0 0}, upload=82914]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:45:51.096210372 (94h14m36.558302856s) [upload=82914]: Done uploading segments... Sep 27 11:45:51.096247213 (94h14m36.558339696s) [upload=82914]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 7.606155938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X....X..................................................X.X] duration: 30.252162859s FAIL (2025-09-27 11:45:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Sep 27 11:45:21.574962251 (94h14m7.037054724s) [upload=82914]: Starting upload Sep 27 11:45:21.575054808 (94h14m7.037147291s) [upload=82914]: Done starting upload Sep 27 11:45:21.575156011 (94h14m7.037248534s) [upload=82914]: Uploading segments... Sep 27 11:45:21.575298221 (94h14m7.037390704s) [upload=82914]: Got next segment. Inline: false Sep 27 11:45:21.575351092 (94h14m7.037443576s) [upload=82914]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:45:21.575356974 (94h14m7.037449447s) [upload=82914]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:45:21.835960057 (94h14m7.298052541s) [seg_pos={0 0}, upload=82914]: Begin upload segment... Sep 27 11:45:21.836030091 (94h14m7.298122564s) [seg_pos={0 0}, upload=82914]: Done begin upload segment. Sep 27 11:45:21.836060489 (94h14m7.298152972s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836070258 (94h14m7.298162731s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.836583224 (94h14m7.298675707s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837000709 (94h14m7.299093192s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.837468460 (94h14m7.299560933s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.839078847 (94h14m7.30117133s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840285635 (94h14m7.302378108s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.840667893 (94h14m7.302760376s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.842333605 (94h14m7.304426098s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.844189488 (94h14m7.306281971s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.845791328 (94h14m7.307883812s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.846640675 (94h14m7.308733158s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.847437573 (94h14m7.309530056s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848493895 (94h14m7.310586378s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.848964320 (94h14m7.311056793s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:21.850072000 (94h14m7.312164483s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Uploading piece... Sep 27 11:45:23.531091188 (94h14m8.993183671s) [upload=82914]: Next returned nil segment Sep 27 11:45:23.531113992 (94h14m8.993206475s) [upload=82914]: Waiting for error group managing segments... Sep 27 11:45:23.811944975 (94h14m9.274037458s) [noise=true , address=145.79.227.3:4151 , piece_id=QPCWGIF5JPQ2N47DMA2KFG3WBVOROLZEIPTQYULNWME253BKHRMQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.820431265 (94h14m9.282523748s) [noise=true , address=145.79.227.11:4732 , piece_id=2EP5DOQFEATCWD2XTIJYSS3PYRFMXD3SQNUS5KFIEE7O7BC7NSUA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.824197606 (94h14m9.286290089s) [noise=true , address=145.79.227.14:4838 , piece_id=2MGSBBWJWP6JIRNPXJHQE7MYGZNPIKISWTMYJ6SFH2AEPJ5RYJ7Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.838835901 (94h14m9.300928374s) [noise=true , address=145.79.227.17:4971 , piece_id=A2ZL6K74JHFRHIJOVUVB7IOMQRWGEWJAIPSYXRRZX3E2EI63JPHA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845110639 (94h14m9.307203122s) [noise=true , address=145.79.227.3:4149 , piece_id=G6LHHIPYLOPFCDQQX53Y34ZRNSEEW73MPCKREDXXUXKM34WBDXXQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.845609919 (94h14m9.307702402s) [noise=true , address=145.79.227.4:4186 , piece_id=SOXJI2F6KOUOKF6UXZKM6KM3IUDUKL4IUR7IWPRLI4UHU3DOFA6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.848133685 (94h14m9.310226168s) [noise=true , address=145.79.227.19:4077 , piece_id=AGD5SGIZZDC3DGW2BJ47FKOMK7AF6OJVOVJE67WPVCJDFDY3WFZQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.964416249 (94h14m9.426508732s) [noise=true , address=45.91.116.7:4010 , piece_id=45IUZAPAS2KOZVOD347KSCKOPB5FRXP2PECGMJRYB3DQ7P4GA6SQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.966287362 (94h14m9.428379845s) [noise=true , address=45.91.116.11:4118 , piece_id=7EXDLL7EZUHFGC72STRX6XBYKD5FU36D5BLE7SDJ67SK53MJPJYQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.970573033 (94h14m9.432665506s) [noise=true , address=45.91.116.13:4153 , piece_id=TYKEFKQXME5425TJPRGOGKQZKZKYBIYQ7UZEYGLYKWSYAAO3GM4Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:23.979664915 (94h14m9.441757398s) [noise=true , address=45.91.116.4:4939 , piece_id=TDQUWWFTWGYBQ4UEFEMLKKVBAPSUO6GNON6533FD6KOVUSNSRFLA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: <nil> Sep 27 11:45:50.846870042 (94h14m36.308962525s) [noise=true , address=77.90.31.19:4842 , piece_id=3NVEDLH6DBHNSRVKMNHVD6PFHODBZFSI44RG2Y7LJHJZAK4LZ2ZA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, address:77.90.31.19:4842): protocol: rpc: dial tcp 77.90.31.19:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.846936599 (94h14m36.309029102s) [noise=true , address=77.90.31.13:4627 , piece_id=EO73PEHUG24GOZGF7MONBRNUYDHH6XOMCH3JM3G5DQXJNJVK4LOA, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:37436->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847116211 (94h14m36.309208694s) [noise=true , address=77.90.31.17:4772 , piece_id=2NXV2PQOBXG2ME74S4QC6KVRO4EFT6YJQJEPPYE6BA36GCGAAINQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:59280->77.90.31.17:4772: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847189230 (94h14m36.309281713s) [noise=true , address=77.90.31.15:4697 , piece_id=OU6OBAYPA4FIJHUL7QVY7QHWKDFI77KSAFMV5UIZ3RE4WRICXEGQ, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 188.166.225.170:47474->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:45:50.847343314 (94h14m36.309435797s) [noise=true , address=77.90.31.10:4520 , piece_id=ZRI7S6UMNFGS6E6QZO46ZTN2PQOSRAZ3EMOEJLLVA4QJ2REIFK6Q, seg_pos={0 0}, upload=82914]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 188.166.225.170:40326->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:45:50.847384322 (94h14m36.309476795s) [seg_pos={0 0}, upload=82914]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Sep 27 11:45:51.096210372 (94h14m36.558302856s) [upload=82914]: Done uploading segments... Sep 27 11:45:51.096247213 (94h14m36.558339696s) [upload=82914]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.25219629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.203156186s FAIL (2025-09-27 11:44:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.20317923s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 46.703667021s FAIL (2025-09-27 11:44:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 46.703704833s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X....X..................................................X.] duration: 9.904686938s FAIL (2025-09-27 11:44:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 11:43:58.909679588 (94h12m44.371772071s) [upload=82891]: Starting upload Sep 27 11:43:58.909776713 (94h12m44.371869196s) [upload=82891]: Done starting upload Sep 27 11:43:58.909861485 (94h12m44.371953968s) [upload=82891]: Uploading segments... Sep 27 11:43:58.910078087 (94h12m44.372170571s) [upload=82891]: Got next segment. Inline: false Sep 27 11:43:58.910121470 (94h12m44.372213943s) [upload=82891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:43:58.910127701 (94h12m44.372220175s) [upload=82891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:43:59.181720612 (94h12m44.643813095s) [seg_pos={0 0}, upload=82891]: Begin upload segment... Sep 27 11:43:59.181834569 (94h12m44.643927042s) [seg_pos={0 0}, upload=82891]: Done begin upload segment. Sep 27 11:43:59.181880186 (94h12m44.643972669s) [noise=true , address=145.79.227.16:4909 , piece_id=GWQXRLN5HGTYFHCNDGZSPUFJ25RYYETVDLYUFW3Z7GEKI6DMYHGQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.181886619 (94h12m44.643979102s) [noise=true , address=77.90.31.19:4842 , piece_id=2IEXNCWEV2E7QH3NABHR3WJAAZQJSSVR6JCF2WXBVC6KXWF4WLEQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.182461373 (94h12m44.644553846s) [noise=true , address=145.79.227.19:4077 , piece_id=GB4IMFI5XCJGET2KZXSYJIMMP7MIYFZI3NHWMI2ILPE7SKHQA5SQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.182852758 (94h12m44.644945241s) [noise=true , address=45.91.116.10:4083 , piece_id=4GKMRFW2JMD7ECR46X4YU64AT2GXHTU56LN7CFK2CGIWREP5J3QA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.183250085 (94h12m44.645342568s) [noise=true , address=77.90.31.21:4878 , piece_id=T2RLW4SNQXB57O6OCUIQPTTPJIWGBXSSIAR26NEKFKUNJZM4MEMA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.183680815 (94h12m44.645773288s) [noise=true , address=145.79.227.3:4149 , piece_id=TKVD2A73WI6LIUF6WPX53CUYFI62XCFVMAJKEJD56H352W377MEQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.184137684 (94h12m44.646230167s) [noise=true , address=45.91.116.9:4261 , piece_id=OPHBMAD7SKIS2IQBN3ZFXVAASVITLCLZYIIQ5BLON43QQNGISKQA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.186036020 (94h12m44.648128503s) [noise=true , address=77.90.31.10:4520 , piece_id=VA3QVGOT75KK32JRBIEVPROC4Y22NXFEL7UMGO4NRHJXRASFSQAQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.186807308 (94h12m44.648899791s) [noise=true , address=45.91.116.4:4938 , piece_id=T66ZO4OYB567WIA5SVNDFOCWPNY4XSBPNBXOFUQFX4JVA6IXHJZQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.188525980 (94h12m44.650618463s) [noise=true , address=45.91.116.8:4045 , piece_id=H4GXLXY4LDGC6TBGANJJJJN57ITVZKDZAAZLDRS3IZATGW6C7QGA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.190425307 (94h12m44.652517791s) [noise=true , address=77.90.31.10:4518 , piece_id=MEBESTBNSFO5SKLTHWVV4CLKX45G67VKYKUUILDLXXWU7Z4XEPNQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.192030163 (94h12m44.654122646s) [noise=true , address=45.91.116.6:4973 , piece_id=IRO7LQ2RSSWOQ2SDUS6EN4Z3KJYAEENOVK4UIKRVACYGQ4N2L7FA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.193582029 (94h12m44.655674512s) [noise=true , address=145.79.227.7:4294 , piece_id=4QZHIUWP4Y2GT3XSHYT76UFGZDTO77EF7LO4N3Y25E24F3ZD4KKQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.195203626 (94h12m44.657296099s) [noise=true , address=145.79.227.3:4150 , piece_id=6QLQBD6PLEOUWH7OQYOMJP3YMMON6LI2RW6EUCNBUMMT6IKQJAIQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.196347073 (94h12m44.658439546s) [noise=true , address=77.90.31.14:4661 , piece_id=OOANPKZUTIXGOEIXNA35KFKZNGMLAEWYBQVU75PZB5SRZO7JR5VA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.199521379 (94h12m44.661613862s) [noise=true , address=45.91.116.12:4227 , piece_id=6WRVTYNIBLWEKQ5SX4QEUYARER5ZZ3IEKDCGILUN6L3YXOARCDUA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:44:00.887126307 (94h12m46.34921879s) [upload=82891]: Next returned nil segment Sep 27 11:44:00.887158739 (94h12m46.349251212s) [upload=82891]: Waiting for error group managing segments... Sep 27 11:44:01.173898103 (94h12m46.635990586s) [noise=true , address=145.79.227.19:4077 , piece_id=GB4IMFI5XCJGET2KZXSYJIMMP7MIYFZI3NHWMI2ILPE7SKHQA5SQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.186092884 (94h12m46.648185387s) [noise=true , address=145.79.227.3:4150 , piece_id=6QLQBD6PLEOUWH7OQYOMJP3YMMON6LI2RW6EUCNBUMMT6IKQJAIQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.193547420 (94h12m46.655639903s) [noise=true , address=145.79.227.7:4294 , piece_id=4QZHIUWP4Y2GT3XSHYT76UFGZDTO77EF7LO4N3Y25E24F3ZD4KKQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.197696490 (94h12m46.659788973s) [noise=true , address=145.79.227.16:4909 , piece_id=GWQXRLN5HGTYFHCNDGZSPUFJ25RYYETVDLYUFW3Z7GEKI6DMYHGQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.214421287 (94h12m46.67651377s) [noise=true , address=145.79.227.3:4149 , piece_id=TKVD2A73WI6LIUF6WPX53CUYFI62XCFVMAJKEJD56H352W377MEQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.398650954 (94h12m46.860743437s) [noise=true , address=77.90.31.19:4842 , piece_id=2IEXNCWEV2E7QH3NABHR3WJAAZQJSSVR6JCF2WXBVC6KXWF4WLEQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.477558248 (94h12m46.939650732s) [noise=true , address=77.90.31.21:4878 , piece_id=T2RLW4SNQXB57O6OCUIQPTTPJIWGBXSSIAR26NEKFKUNJZM4MEMA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.531227955 (94h12m46.993320438s) [noise=true , address=77.90.31.10:4520 , piece_id=VA3QVGOT75KK32JRBIEVPROC4Y22NXFEL7UMGO4NRHJXRASFSQAQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.550649828 (94h12m47.012742321s) [noise=true , address=77.90.31.10:4518 , piece_id=MEBESTBNSFO5SKLTHWVV4CLKX45G67VKYKUUILDLXXWU7Z4XEPNQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.875090042 (94h12m47.337182525s) [noise=true , address=77.90.31.14:4661 , piece_id=OOANPKZUTIXGOEIXNA35KFKZNGMLAEWYBQVU75PZB5SRZO7JR5VA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:28.195460733 (94h13m13.657553216s) [noise=true , address=45.91.116.10:4083 , piece_id=4GKMRFW2JMD7ECR46X4YU64AT2GXHTU56LN7CFK2CGIWREP5J3QA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48518->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195574008 (94h13m13.657666501s) [noise=true , address=45.91.116.8:4045 , piece_id=H4GXLXY4LDGC6TBGANJJJJN57ITVZKDZAAZLDRS3IZATGW6C7QGA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195643410 (94h13m13.657735883s) [noise=true , address=45.91.116.12:4227 , piece_id=6WRVTYNIBLWEKQ5SX4QEUYARER5ZZ3IEKDCGILUN6L3YXOARCDUA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195780982 (94h13m13.657873485s) [noise=true , address=45.91.116.6:4973 , piece_id=IRO7LQ2RSSWOQ2SDUS6EN4Z3KJYAEENOVK4UIKRVACYGQ4N2L7FA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:52898->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195856817 (94h13m13.6579493s) [noise=true , address=45.91.116.9:4261 , piece_id=OPHBMAD7SKIS2IQBN3ZFXVAASVITLCLZYIIQ5BLON43QQNGISKQA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:38412->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.196025568 (94h13m13.658118061s) [noise=true , address=45.91.116.4:4938 , piece_id=T66ZO4OYB567WIA5SVNDFOCWPNY4XSBPNBXOFUQFX4JVA6IXHJZQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:44:28.196056076 (94h13m13.658148559s) [seg_pos={0 0}, upload=82891]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 11:44:28.452903361 (94h13m13.914995834s) [upload=82891]: Done uploading segments... Sep 27 11:44:28.452933980 (94h13m13.915026463s) [upload=82891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 9.90470866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X....X..................................................X] duration: 30.259466983s FAIL (2025-09-27 11:44:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== Sep 27 11:43:58.909679588 (94h12m44.371772071s) [upload=82891]: Starting upload Sep 27 11:43:58.909776713 (94h12m44.371869196s) [upload=82891]: Done starting upload Sep 27 11:43:58.909861485 (94h12m44.371953968s) [upload=82891]: Uploading segments... Sep 27 11:43:58.910078087 (94h12m44.372170571s) [upload=82891]: Got next segment. Inline: false Sep 27 11:43:58.910121470 (94h12m44.372213943s) [upload=82891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:43:58.910127701 (94h12m44.372220175s) [upload=82891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:43:59.181720612 (94h12m44.643813095s) [seg_pos={0 0}, upload=82891]: Begin upload segment... Sep 27 11:43:59.181834569 (94h12m44.643927042s) [seg_pos={0 0}, upload=82891]: Done begin upload segment. Sep 27 11:43:59.181880186 (94h12m44.643972669s) [noise=true , address=145.79.227.16:4909 , piece_id=GWQXRLN5HGTYFHCNDGZSPUFJ25RYYETVDLYUFW3Z7GEKI6DMYHGQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.181886619 (94h12m44.643979102s) [noise=true , address=77.90.31.19:4842 , piece_id=2IEXNCWEV2E7QH3NABHR3WJAAZQJSSVR6JCF2WXBVC6KXWF4WLEQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.182461373 (94h12m44.644553846s) [noise=true , address=145.79.227.19:4077 , piece_id=GB4IMFI5XCJGET2KZXSYJIMMP7MIYFZI3NHWMI2ILPE7SKHQA5SQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.182852758 (94h12m44.644945241s) [noise=true , address=45.91.116.10:4083 , piece_id=4GKMRFW2JMD7ECR46X4YU64AT2GXHTU56LN7CFK2CGIWREP5J3QA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.183250085 (94h12m44.645342568s) [noise=true , address=77.90.31.21:4878 , piece_id=T2RLW4SNQXB57O6OCUIQPTTPJIWGBXSSIAR26NEKFKUNJZM4MEMA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.183680815 (94h12m44.645773288s) [noise=true , address=145.79.227.3:4149 , piece_id=TKVD2A73WI6LIUF6WPX53CUYFI62XCFVMAJKEJD56H352W377MEQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.184137684 (94h12m44.646230167s) [noise=true , address=45.91.116.9:4261 , piece_id=OPHBMAD7SKIS2IQBN3ZFXVAASVITLCLZYIIQ5BLON43QQNGISKQA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.186036020 (94h12m44.648128503s) [noise=true , address=77.90.31.10:4520 , piece_id=VA3QVGOT75KK32JRBIEVPROC4Y22NXFEL7UMGO4NRHJXRASFSQAQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.186807308 (94h12m44.648899791s) [noise=true , address=45.91.116.4:4938 , piece_id=T66ZO4OYB567WIA5SVNDFOCWPNY4XSBPNBXOFUQFX4JVA6IXHJZQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.188525980 (94h12m44.650618463s) [noise=true , address=45.91.116.8:4045 , piece_id=H4GXLXY4LDGC6TBGANJJJJN57ITVZKDZAAZLDRS3IZATGW6C7QGA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.190425307 (94h12m44.652517791s) [noise=true , address=77.90.31.10:4518 , piece_id=MEBESTBNSFO5SKLTHWVV4CLKX45G67VKYKUUILDLXXWU7Z4XEPNQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.192030163 (94h12m44.654122646s) [noise=true , address=45.91.116.6:4973 , piece_id=IRO7LQ2RSSWOQ2SDUS6EN4Z3KJYAEENOVK4UIKRVACYGQ4N2L7FA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.193582029 (94h12m44.655674512s) [noise=true , address=145.79.227.7:4294 , piece_id=4QZHIUWP4Y2GT3XSHYT76UFGZDTO77EF7LO4N3Y25E24F3ZD4KKQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.195203626 (94h12m44.657296099s) [noise=true , address=145.79.227.3:4150 , piece_id=6QLQBD6PLEOUWH7OQYOMJP3YMMON6LI2RW6EUCNBUMMT6IKQJAIQ, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.196347073 (94h12m44.658439546s) [noise=true , address=77.90.31.14:4661 , piece_id=OOANPKZUTIXGOEIXNA35KFKZNGMLAEWYBQVU75PZB5SRZO7JR5VA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:43:59.199521379 (94h12m44.661613862s) [noise=true , address=45.91.116.12:4227 , piece_id=6WRVTYNIBLWEKQ5SX4QEUYARER5ZZ3IEKDCGILUN6L3YXOARCDUA, seg_pos={0 0}, upload=82891]: Uploading piece... Sep 27 11:44:00.887126307 (94h12m46.34921879s) [upload=82891]: Next returned nil segment Sep 27 11:44:00.887158739 (94h12m46.349251212s) [upload=82891]: Waiting for error group managing segments... Sep 27 11:44:01.173898103 (94h12m46.635990586s) [noise=true , address=145.79.227.19:4077 , piece_id=GB4IMFI5XCJGET2KZXSYJIMMP7MIYFZI3NHWMI2ILPE7SKHQA5SQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.186092884 (94h12m46.648185387s) [noise=true , address=145.79.227.3:4150 , piece_id=6QLQBD6PLEOUWH7OQYOMJP3YMMON6LI2RW6EUCNBUMMT6IKQJAIQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.193547420 (94h12m46.655639903s) [noise=true , address=145.79.227.7:4294 , piece_id=4QZHIUWP4Y2GT3XSHYT76UFGZDTO77EF7LO4N3Y25E24F3ZD4KKQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.197696490 (94h12m46.659788973s) [noise=true , address=145.79.227.16:4909 , piece_id=GWQXRLN5HGTYFHCNDGZSPUFJ25RYYETVDLYUFW3Z7GEKI6DMYHGQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.214421287 (94h12m46.67651377s) [noise=true , address=145.79.227.3:4149 , piece_id=TKVD2A73WI6LIUF6WPX53CUYFI62XCFVMAJKEJD56H352W377MEQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.398650954 (94h12m46.860743437s) [noise=true , address=77.90.31.19:4842 , piece_id=2IEXNCWEV2E7QH3NABHR3WJAAZQJSSVR6JCF2WXBVC6KXWF4WLEQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.477558248 (94h12m46.939650732s) [noise=true , address=77.90.31.21:4878 , piece_id=T2RLW4SNQXB57O6OCUIQPTTPJIWGBXSSIAR26NEKFKUNJZM4MEMA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.531227955 (94h12m46.993320438s) [noise=true , address=77.90.31.10:4520 , piece_id=VA3QVGOT75KK32JRBIEVPROC4Y22NXFEL7UMGO4NRHJXRASFSQAQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.550649828 (94h12m47.012742321s) [noise=true , address=77.90.31.10:4518 , piece_id=MEBESTBNSFO5SKLTHWVV4CLKX45G67VKYKUUILDLXXWU7Z4XEPNQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:01.875090042 (94h12m47.337182525s) [noise=true , address=77.90.31.14:4661 , piece_id=OOANPKZUTIXGOEIXNA35KFKZNGMLAEWYBQVU75PZB5SRZO7JR5VA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: <nil> Sep 27 11:44:28.195460733 (94h13m13.657553216s) [noise=true , address=45.91.116.10:4083 , piece_id=4GKMRFW2JMD7ECR46X4YU64AT2GXHTU56LN7CFK2CGIWREP5J3QA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, address:45.91.116.10:4083): protocol: write tcp 188.166.225.170:48518->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195574008 (94h13m13.657666501s) [noise=true , address=45.91.116.8:4045 , piece_id=H4GXLXY4LDGC6TBGANJJJJN57ITVZKDZAAZLDRS3IZATGW6C7QGA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: rpc: dial tcp 45.91.116.8:4045: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195643410 (94h13m13.657735883s) [noise=true , address=45.91.116.12:4227 , piece_id=6WRVTYNIBLWEKQ5SX4QEUYARER5ZZ3IEKDCGILUN6L3YXOARCDUA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195780982 (94h13m13.657873485s) [noise=true , address=45.91.116.6:4973 , piece_id=IRO7LQ2RSSWOQ2SDUS6EN4Z3KJYAEENOVK4UIKRVACYGQ4N2L7FA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: write tcp 188.166.225.170:52898->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.195856817 (94h13m13.6579493s) [noise=true , address=45.91.116.9:4261 , piece_id=OPHBMAD7SKIS2IQBN3ZFXVAASVITLCLZYIIQ5BLON43QQNGISKQA, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, address:45.91.116.9:4261): protocol: write tcp 188.166.225.170:38412->45.91.116.9:4261: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:44:28.196025568 (94h13m13.658118061s) [noise=true , address=45.91.116.4:4938 , piece_id=T66ZO4OYB567WIA5SVNDFOCWPNY4XSBPNBXOFUQFX4JVA6IXHJZQ, seg_pos={0 0}, upload=82891]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Sep 27 11:44:28.196056076 (94h13m13.658148559s) [seg_pos={0 0}, upload=82891]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Sep 27 11:44:28.452903361 (94h13m13.914995834s) [upload=82891]: Done uploading segments... Sep 27 11:44:28.452933980 (94h13m13.915026463s) [upload=82891]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.259499706s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 7.390524285s FAIL (2025-09-27 11:43:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.390552919s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X....X..................................................] duration: 6.209070552s FAIL (2025-09-27 11:42:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.209100949s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.893899225s FAIL (2025-09-27 11:42:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.89392869s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 37.917092944s FAIL (2025-09-27 11:42:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 37.917124094s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X....X.................................................] duration: 7.052770656s FAIL (2025-09-27 11:41:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.052800884s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X....X................................................] duration: 6.544667546s FAIL (2025-09-27 11:41:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.544700839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X....X...............................................] duration: 6.070033144s FAIL (2025-09-27 11:41:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.070072419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 10.296950602s FAIL (2025-09-27 11:40:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 10.297018862s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X....X..............................................] duration: 6.21179659s FAIL (2025-09-27 11:40:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.211832969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.746313785s FAIL (2025-09-27 11:40:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.746344804s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X....X.............................................] duration: 9.792541375s FAIL (2025-09-27 11:39:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 9.792572313s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.13481845s FAIL (2025-09-27 11:39:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.134850972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 5.938447273s FAIL (2025-09-27 11:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.938519661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 7.635114789s FAIL (2025-09-27 11:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.635150686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.700821799s FAIL (2025-09-27 11:38:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.700856305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 6.101082088s FAIL (2025-09-27 11:37:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.101129077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.782947631s FAIL (2025-09-27 11:37:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.782985704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 6.153232521s FAIL (2025-09-27 11:36:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.153262067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.68962541s FAIL (2025-09-27 11:36:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.689660527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 6.398298615s FAIL (2025-09-27 11:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.398355174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.720564483s FAIL (2025-09-27 11:35:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.720611083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 6.841050863s FAIL (2025-09-27 11:35:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.841083886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.812195536s FAIL (2025-09-27 11:35:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.812220454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 6.010400976s FAIL (2025-09-27 11:34:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.010434339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 5.775042931s FAIL (2025-09-27 11:34:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775075363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.785772372s FAIL (2025-09-27 11:34:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.785816657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 6.570112959s FAIL (2025-09-27 11:33:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.570144038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.817378886s FAIL (2025-09-27 11:33:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.817429483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 5.896589068s FAIL (2025-09-27 11:32:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.896628353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 6.69285766s FAIL (2025-09-27 11:32:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.692887087s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)