Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.13481845s FAIL (2025-09-27 11:39:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.134850972s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X....X............................................] duration: 5.938447273s FAIL (2025-09-27 11:38:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.938519661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X....X...........................................] duration: 7.635114789s FAIL (2025-09-27 11:38:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.635150686s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.700821799s FAIL (2025-09-27 11:38:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.700856305s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X....X..........................................] duration: 6.101082088s FAIL (2025-09-27 11:37:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.101129077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.782947631s FAIL (2025-09-27 11:37:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.782985704s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X....X.........................................] duration: 6.153232521s FAIL (2025-09-27 11:36:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.153262067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.68962541s FAIL (2025-09-27 11:36:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.689660527s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X....X........................................] duration: 6.398298615s FAIL (2025-09-27 11:36:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.398355174s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.720564483s FAIL (2025-09-27 11:35:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.720611083s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X....X.......................................] duration: 6.841050863s FAIL (2025-09-27 11:35:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.841083886s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.812195536s FAIL (2025-09-27 11:35:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.812220454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X....X......................................] duration: 6.010400976s FAIL (2025-09-27 11:34:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.010434339s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X....X.....................................] duration: 5.775042931s FAIL (2025-09-27 11:34:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.775075363s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.785772372s FAIL (2025-09-27 11:34:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.785816657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X....X....................................] duration: 6.570112959s FAIL (2025-09-27 11:33:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.570144038s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.817378886s FAIL (2025-09-27 11:33:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.817429483s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X....X...................................] duration: 5.896589068s FAIL (2025-09-27 11:32:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.896628353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X....X..................................] duration: 6.69285766s FAIL (2025-09-27 11:32:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.692887087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.825950477s FAIL (2025-09-27 11:32:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.825987587s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X....X.................................] duration: 6.294757646s FAIL (2025-09-27 11:31:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.294789438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.134117443s FAIL (2025-09-27 11:31:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.134153511s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X....X................................] duration: 6.524655234s FAIL (2025-09-27 11:31:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.524688276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.906676474s FAIL (2025-09-27 11:30:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.906713685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X....X...............................] duration: 6.042101846s FAIL (2025-09-27 11:30:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.042158286s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.977724502s FAIL (2025-09-27 11:30:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.97776095s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.643629055s FAIL (2025-09-27 11:29:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.643658812s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X....X..............................] duration: 6.35437579s FAIL (2025-09-27 11:29:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.354406399s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.99438672s FAIL (2025-09-27 11:29:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.994424762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X....X.............................] duration: 5.965432405s FAIL (2025-09-27 11:28:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965473734s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.886777924s FAIL (2025-09-27 11:28:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.886809362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.727528655s FAIL (2025-09-27 11:27:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.727558192s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X....X............................] duration: 5.965119239s FAIL (2025-09-27 11:27:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.965151291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.793522409s FAIL (2025-09-27 11:26:42 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.793552727s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X....X...........................] duration: 21.923840602s FAIL (2025-09-27 11:26:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 21.923874317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.............................................X.............] duration: 5.754702017s FAIL (2025-09-27 11:25:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.75473462s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.............................................X............] duration: 38.231118187s FAIL (2025-09-27 11:25:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 38.231152433s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X....X..........................] duration: 13.875389565s FAIL (2025-09-27 11:25:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.875421677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X....X.........................] duration: 6.176085025s FAIL (2025-09-27 11:24:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.176138567s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X....X........................] duration: 7.174278297s FAIL (2025-09-27 11:23:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 7.17432712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.............................................X...........] duration: 7.633849314s FAIL (2025-09-27 11:23:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.63388364s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X....X.......................] duration: 6.033547158s FAIL (2025-09-27 11:23:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.033584279s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.............................................X..........] duration: 6.132134685s FAIL (2025-09-27 11:22:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.132165944s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X....X......................] duration: 5.998187848s FAIL (2025-09-27 11:22:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.998221563s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.............................................X.........] duration: 6.012203174s FAIL (2025-09-27 11:21:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.012227601s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X....X.....................] duration: 14.60947984s FAIL (2025-09-27 11:21:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.609512433s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.............................................X........] duration: 6.050981406s FAIL (2025-09-27 11:20:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.051014108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.................................X....X....................] duration: 6.25508286s FAIL (2025-09-27 11:20:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.255115582s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.................................X....X...................] duration: 6.463988542s FAIL (2025-09-27 11:20:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 27 11:05:11.660039662 (93h33m57.122132145s) [upload=82327]: Starting upload Sep 27 11:05:11.660139762 (93h33m57.122232235s) [upload=82327]: Done starting upload Sep 27 11:05:11.660241336 (93h33m57.122333809s) [upload=82327]: Uploading segments... Sep 27 11:05:11.660555815 (93h33m57.122648298s) [upload=82327]: Got next segment. Inline: false Sep 27 11:05:11.660590501 (93h33m57.122682974s) [upload=82327]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:05:11.660595149 (93h33m57.122687622s) [upload=82327]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:05:11.919775543 (93h33m57.381868026s) [seg_pos={0 0}, upload=82327]: Begin upload segment... Sep 27 11:05:11.919843092 (93h33m57.381935575s) [seg_pos={0 0}, upload=82327]: Done begin upload segment. Sep 27 11:05:11.919873159 (93h33m57.381965642s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.919885983 (93h33m57.381978456s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920417535 (93h33m57.382510018s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920609321 (93h33m57.382701804s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920829430 (93h33m57.382921913s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.920953336 (93h33m57.383045819s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921339171 (93h33m57.383431654s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921411739 (93h33m57.383504222s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921741446 (93h33m57.383833929s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.921837810 (93h33m57.383930283s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922131269 (93h33m57.384223752s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922145446 (93h33m57.384237929s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922499800 (93h33m57.384592283s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922655277 (93h33m57.38474776s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.922981247 (93h33m57.38507373s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:11.923096106 (93h33m57.385188589s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Uploading piece... Sep 27 11:05:13.626049378 (93h33m59.088141861s) [upload=82327]: Next returned nil segment Sep 27 11:05:13.626074577 (93h33m59.08816706s) [upload=82327]: Waiting for error group managing segments... Sep 27 11:05:13.912337911 (93h33m59.374430394s) [noise=true , address=145.79.227.6:4257 , piece_id=ADYURADFMUZBXWEF2PMK36NA7HKQVUUWJVZV2N37XMOXXANMSW7Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.912546749 (93h33m59.374639222s) [noise=true , address=145.79.227.3:4151 , piece_id=5N4RUUEVQGWJCR3PDLPZ2VBNZFADEVXE4BS5OKMEVPVNXDIL5BSA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.922830701 (93h33m59.384923184s) [noise=true , address=145.79.227.13:4801 , piece_id=B4FMP3G5BTSB7LHPYGJ26KT2VOMQEFK6DIDWDAUVXM3HP4FP7SRQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:13.926632961 (93h33m59.388725454s) [noise=true , address=145.79.227.4:4187 , piece_id=5GWEPWTIUCQ6N7O2RY4UEBEQHFLMEMVTAH5HAYJMRUJ6W6MOZ27A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.317327920 (93h33m59.779420403s) [noise=true , address=45.91.116.14:4189 , piece_id=CNFGRJZOU3U24EG5P3OHTK442WDC4SRMT3ZL5CRSV7LY6YWO6DWA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.324787555 (93h33m59.786880038s) [noise=true , address=45.91.116.11:4118 , piece_id=NVUNLKYYGA2LEJ3J3XMNRDKRF4QFZ2YT5KUIMZYK5QEDWGFGFGRA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.334850216 (93h33m59.796942699s) [noise=true , address=45.91.116.5:4299 , piece_id=ZKYAAWWO2NJEUFKLD6YGOQHCADROKRWTWHV6ZOCMPUV75KRSXNXQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.355251704 (93h33m59.817344207s) [noise=true , address=45.91.116.9:4263 , piece_id=SIGAAG3PRPYGKFIANWSHC6HEPU2HQFNZHWUMOOOQT4RWITVV56RQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.372461594 (93h33m59.834554077s) [noise=true , address=45.91.116.6:4975 , piece_id=GYXNUIDATY3C3CE5AWB7IKOGB6TQ47WMMIZ7KX7BUVQWENVL6QHQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.430016488 (93h33m59.892108971s) [noise=true , address=45.91.116.12:4227 , piece_id=ELF55PIBFUE2QBTLVWWIVAGSIZRFSSYQ2VNQ7YFBLUYMYY3QKRYQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.446961745 (93h33m59.909054218s) [noise=true , address=45.91.116.13:4155 , piece_id=2H3B3QRSSAAUGY22N7RIIBJNSKUJVQPXOLSN32U6OC2UDICKZ47A, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:14.723317481 (93h34m0.185409964s) [noise=true , address=45.91.116.9:4261 , piece_id=OFZSG6IF56BP4JEPMF62W6NSAKSSS4LU6X76VXI2YXW5U7KDPGZQ, seg_pos={0 0}, upload=82327]: Done uploading piece. err: <nil> Sep 27 11:05:40.951601776 (93h34m26.413694259s) [noise=true , address=77.90.31.21:4877 , piece_id=HGVQ76WWGAEAPRRHIUX2JOGRGEUNGMI3NXQRAU7GS3LBNL7OOVMA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: context deadline exceeded; context deadline exceeded Sep 27 11:05:40.951748246 (93h34m26.413840739s) [noise=true , address=77.90.31.18:4807 , piece_id=MJ33XHIF2MWKSGYBP77OCK7BXDRN6UYOLAUOB6MPBSN7WOY34UDA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951794593 (93h34m26.413887077s) [noise=true , address=77.90.31.15:4698 , piece_id=AZDLHMV2L7DJGRZCT5AO4LJKOANOYLCYXJMXIRPXCVYOZJWIJT6Q, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951823919 (93h34m26.413916392s) [noise=true , address=77.90.31.10:4517 , piece_id=V33RUGUK6HYKIBCHOWKBXS2ORRQ5S6ZTNLODZLNBFK2GUXLF7BEA, seg_pos={0 0}, upload=82327]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 188.166.225.170:54768->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:05:40.951851973 (93h34m26.413944446s) [seg_pos={0 0}, upload=82327]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 27 11:05:41.196814002 (93h34m26.658906485s) [upload=82327]: Done uploading segments... Sep 27 11:05:41.196848748 (93h34m26.658941231s) [upload=82327]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.4640662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.............................................X.......] duration: 6.436388978s FAIL (2025-09-27 11:20:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== : Uploading segments... Sep 27 11:14:58.287300286 (93h43m43.74939277s) [upload=82467]: Got next segment. Inline: false Sep 27 11:14:58.287341715 (93h43m43.749434188s) [upload=82467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 27 11:14:58.287345733 (93h43m43.749438206s) [upload=82467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 27 11:14:58.544874210 (93h43m44.006966683s) [seg_pos={0 0}, upload=82467]: Begin upload segment... Sep 27 11:14:58.544951808 (93h43m44.007044321s) [seg_pos={0 0}, upload=82467]: Done begin upload segment. Sep 27 11:14:58.544996182 (93h43m44.007088715s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545004248 (93h43m44.007096721s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545613318 (93h43m44.007705811s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.545911065 (93h43m44.008003538s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546194014 (93h43m44.008286487s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546459208 (93h43m44.008551681s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546712100 (93h43m44.008804583s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.546973878 (93h43m44.009066361s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547259863 (93h43m44.009352346s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547544045 (93h43m44.009636528s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.547845578 (93h43m44.009938051s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.549130555 (93h43m44.011223038s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.550449877 (93h43m44.01254235s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.551652678 (93h43m44.013745181s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.552857592 (93h43m44.014950065s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.554096612 (93h43m44.016189085s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.555207938 (93h43m44.017300411s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:14:58.558938952 (93h43m44.021031435s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Uploading piece... Sep 27 11:15:00.008902438 (93h43m45.470994931s) [upload=82467]: Next returned nil segment Sep 27 11:15:00.008925943 (93h43m45.471018426s) [upload=82467]: Waiting for error group managing segments... Sep 27 11:15:00.395996191 (93h43m45.858088665s) [noise=true , address=145.79.227.2:4144 , piece_id=4PYXMR3WJNRFLVWIZIOEXFCPHAV3GDY6GX6I6HSCV26656DH4VNA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.397927739 (93h43m45.860020222s) [noise=true , address=145.79.227.52:4470 , piece_id=IDHWXJJCM2JFGXAZBG3THX6TRXWRJGGA4CKXCOUVWFSFS7GNNG3A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.402986712 (93h43m45.865079185s) [noise=true , address=145.79.227.42:4134 , piece_id=LKZMY2IXRKSYA2N5TECXCJHXG3GLB6KBSBC2C6QEGW2R2MJDC4OA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.407543098 (93h43m45.869635571s) [noise=true , address=145.79.227.40:4070 , piece_id=Y3FG5STWTTRWVF7HASGHWV3MV4H2IFDKKEL5L2ZPRG6L6AABIEDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.414196637 (93h43m45.876289121s) [noise=true , address=145.79.227.21:4477 , piece_id=CSLX3TKQ26US34XN3OVOBLD24QLLVNO7HP5OCJW2YFMLZZOMXRXA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.417474649 (93h43m45.879567122s) [noise=true , address=145.79.227.10:4436 , piece_id=F4RAWBKYNZTNJZ6RD5KYT6DAVTAX3J52DB5JGG4VKGGNZSZD7P4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.456226509 (93h43m45.918318982s) [noise=true , address=145.79.227.4:4203 , piece_id=MGZCTU6Y3K7CUQIP3PAVEBM7NB6UGHOIT626FY5OAFDZXA4RH5KA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.459205813 (93h43m45.921298296s) [noise=true , address=145.79.227.20:4450 , piece_id=SW5V2XBOZPGU257DO7VBWTWK472UJX2BOJ736BVYCL7FQEXWTSDA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.487963482 (93h43m45.950055985s) [noise=true , address=145.79.227.38:4987 , piece_id=ZL3ZB4SHRXBC3M7G4HESLNN5PYAPAXBJ7HCMDRGHXGOVU7HJUZSA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.503245582 (93h43m45.965338065s) [noise=true , address=45.91.116.11:4151 , piece_id=XTRI5PI5U5S5TAORGOCZOMV7C5FCU7Q4XIOXHWNOK3WM3NEWD23Q, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.522156081 (93h43m45.984248564s) [noise=true , address=145.79.227.35:4497 , piece_id=LSLXQFIYL6S32NQC55M74Q4BJVHC2XSJD4GQFWMGYK5MB7RXVV4A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.530216519 (93h43m45.992308993s) [noise=true , address=45.91.116.4:4958 , piece_id=5L5V5UJNMYFMAHLKBFSPYPRRJUQGQV2ZWDNNUBSBAWSR5WJZU42A, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:00.576998491 (93h43m46.039090964s) [noise=true , address=45.91.116.5:4319 , piece_id=VUEGP3UM7MWS6VLD66FYJARCE6TFJGXXVJFCIXZ7GQAJX5ONSMTA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: <nil> Sep 27 11:15:57.579530308 (93h44m43.041622791s) [noise=true , address=77.90.31.19:4844 , piece_id=NYJNNJHMYSQOFWTRJJMKTUUW2MW4LQLNQV4WDAGHGMV7TML2EPJA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, address:77.90.31.19:4844): protocol: write tcp 188.166.225.170:53020->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579572510 (93h44m43.041664993s) [noise=true , address=77.90.31.18:4836 , piece_id=UK3XCZTZIAZMFOMZEEZ45LZ2IY4E2EHBUCUZR2XJKGGK74EHW2LA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12Ga6Vu6VGDGdHP3xxTY9ywyUSXSjo7fybEEzcYcqFbiWGmWENv, address:77.90.31.18:4836): protocol: rpc: dial tcp 77.90.31.18:4836: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579730330 (93h44m43.041822803s) [noise=true , address=77.90.31.15:4700 , piece_id=AOCKKKWNQRUXOD2FMXM64M2AR4QIWJ6QB6MYO67CPS4KR2O5XTRA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, address:77.90.31.15:4700): protocol: write tcp 188.166.225.170:57934->77.90.31.15:4700: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579865928 (93h44m43.041958421s) [noise=true , address=77.90.31.10:4537 , piece_id=H5MTOK5AMQ2YKXVGQC5T4GABSVUKD6OY4I57GWEUENIN6TYZTHFA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:12M5ger8kngGvMCD9c2qBxpFNCpLxka2fh8iWrf2XerxRiQEUja, address:77.90.31.10:4537): protocol: write tcp 188.166.225.170:35064->77.90.31.10:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579938646 (93h44m43.042031129s) [noise=true , address=77.90.31.16:4761 , piece_id=3DGZNEXF4G7MJ5YIH2GQ5N26ZVIGQ63IFN3B6WEKP4BGPQ4TIJQA, seg_pos={0 0}, upload=82467]: Done uploading piece. err: ecclient: upload failed (node:1x6et6pugViEqRsU2Tx7QC4h1CixBcWWEqU6fNBLXwK6f26DZA, address:77.90.31.16:4761): protocol: rpc: dial tcp 77.90.31.16:4761: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 27 11:15:57.579966239 (93h44m43.042058712s) [seg_pos={0 0}, upload=82467]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 27 11:15:57.825018271 (93h44m43.287110744s) [upload=82467]: Done uploading segments... Sep 27 11:15:57.825047537 (93h44m43.28714002s) [upload=82467]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.436416311s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)