Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..XXX..XX..................................................] duration: 5.90816761s FAIL (2025-09-19 20:58:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 5.908201714s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX..X.X..X..XX..............................................] duration: 6.161645162s FAIL (2025-09-19 20:58:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.161671532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 8.385555139s FAIL (2025-09-19 20:58:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 8.385586328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [XXX..X.X..X..XX.............................................] duration: 6.360014596s FAIL (2025-09-19 20:58:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.360047469s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.189692369s FAIL (2025-09-19 20:57:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.189723669s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..XXX..XX.................................................] duration: 6.058756348s FAIL (2025-09-19 20:57:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.058792457s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.........X.................................................] duration: 6.216831568s FAIL (2025-09-19 20:57:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.216862999s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.........X................................................] duration: 6.647998395s FAIL (2025-09-19 20:57:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.648032939s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XXXX..X.X..X..XX............................................] duration: 6.225147043s FAIL (2025-09-19 20:56:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.225172021s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.X..XXX..XX................................................] duration: 6.516026853s FAIL (2025-09-19 20:56:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.51606166s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXXXX..X.X..X..XX...........................................] duration: 6.045829581s FAIL (2025-09-19 20:56:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.045852676s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.X..XXX..XX...............................................] duration: 6.258809749s FAIL (2025-09-19 20:56:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.258837863s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.........X...............................................] duration: 5.97257273s FAIL (2025-09-19 20:55:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.972603056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.X..XXX..XX..............................................] duration: 6.007120695s FAIL (2025-09-19 20:55:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.007172705s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXX..X.X..X..XX..........................................] duration: 6.000452125s FAIL (2025-09-19 20:55:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.000477764s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X.........X..............................................] duration: 6.037729121s FAIL (2025-09-19 20:54:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.037758546s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X.X..XXX..XX.............................................] duration: 6.249063075s FAIL (2025-09-19 20:54:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.249099675s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.XXXXXX..X.X..X..XX.........................................] duration: 6.184534527s FAIL (2025-09-19 20:54:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.184566188s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [XX..X.X..XXX..XX............................................] duration: 6.467853329s FAIL (2025-09-19 20:54:01 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.467886501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XXX..X.X..XXX..XX...........................................] duration: 6.063216081s FAIL (2025-09-19 20:53:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.063255175s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 5.993379508s FAIL (2025-09-19 20:53:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 5.993403704s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..XXXXXX..X.X..X..XX........................................] duration: 6.458661978s FAIL (2025-09-19 20:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.458686264s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [XXXX..X.X..XXX..XX..........................................] duration: 6.207095707s FAIL (2025-09-19 20:53:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.207143579s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 32.037288824s FAIL (2025-09-19 20:53:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 32.037314385s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [XXXXX..X.X..XXX..XX.........................................] duration: 6.755103946s FAIL (2025-09-19 20:52:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.755135746s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 6.707828453s FAIL (2025-09-19 20:52:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.707858852s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...XXXXXX..X.X..X..XX.......................................] duration: 6.048807466s FAIL (2025-09-19 20:52:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.048834648s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXX..X.X..XXX..XX........................................] duration: 6.054019776s FAIL (2025-09-19 20:52:12 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.054054371s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....XXXXXX..X.X..X..XX......................................] duration: 6.255430636s FAIL (2025-09-19 20:51:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.255458239s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 7.142368677s FAIL (2025-09-19 20:51:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.142403353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXX..X.X..XXX..XX.......................................] duration: 6.328789498s FAIL (2025-09-19 20:51:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.328830757s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....XXXXXX..X.X..X..XX.....................................] duration: 6.105403895s FAIL (2025-09-19 20:50:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.105456315s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 6.999478944s FAIL (2025-09-19 20:50:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.999516156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [XXXXXXXX..X.X..XXX..XX......................................] duration: 6.417670392s FAIL (2025-09-19 20:50:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.417700769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [XXXXXXXXX..X.X..XXX..XX.....................................] duration: 6.888356095s FAIL (2025-09-19 20:50:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.888388859s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........X........................................] duration: 6.752944234s FAIL (2025-09-19 20:50:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.75297383s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......XXXXXX..X.X..X..XX....................................] duration: 5.963559585s FAIL (2025-09-19 20:50:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.963589994s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [XXXXXXXXXX..X.X..XXX..XX....................................] duration: 6.543281395s FAIL (2025-09-19 20:49:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.543319537s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 7.643870539s FAIL (2025-09-19 20:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.643902279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.XXXXXXXXXX..X.X..XXX..XX...................................] duration: 6.14071656s FAIL (2025-09-19 20:49:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.140768168s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.......XXXXXX..X.X..X..XX...................................] duration: 6.023137235s FAIL (2025-09-19 20:49:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.023164948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 6.846379545s FAIL (2025-09-19 20:49:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.846416724s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [..XXXXXXXXXX..X.X..XXX..XX..................................] duration: 11.558642105s FAIL (2025-09-19 20:48:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 11.55867621s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [........XXXXXX..X.X..X..XX..................................] duration: 6.273992661s FAIL (2025-09-19 20:48:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.274032919s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........X.....................................] duration: 7.223260869s FAIL (2025-09-19 20:48:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 7.223294191s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 6.126873558s FAIL (2025-09-19 20:48:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.126895369s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X..XXXXXXXXXX..X.X..XXX..XX.................................] duration: 7.345235609s FAIL (2025-09-19 20:47:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 7.345267919s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.........XXXXXX..X.X..X..XX.................................] duration: 6.289359195s FAIL (2025-09-19 20:47:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.28963028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.X..XXXXXXXXXX..X.X..XXX..XX................................] duration: 6.070346304s FAIL (2025-09-19 20:47:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 3) ======LOGS====== 4m37.804485978s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766046937 (318h54m37.805620067s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.766926762 (318h54m37.806499892s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.769042241 (318h54m37.808615371s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.771299770 (318h54m37.81087291s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.772714714 (318h54m37.812287844s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.774261980 (318h54m37.81383511s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.779721416 (318h54m37.819294546s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.781409851 (318h54m37.820983042s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.782697153 (318h54m37.822270383s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.787232208 (318h54m37.826805338s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.788881319 (318h54m37.828454459s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:01.789202410 (318h54m37.82877554s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Uploading piece... Sep 19 20:23:03.544838046 (318h54m39.584411176s) [upload=280330]: Next returned nil segment Sep 19 20:23:03.544863885 (318h54m39.584437015s) [upload=280330]: Waiting for error group managing segments... Sep 19 20:23:03.830180165 (318h54m39.869753295s) [noise=true , address=145.79.227.18:4006 , piece_id=C4ABLWUA366CIAMI5A3JSJZXTOAC5YCCEFT235HV3X7X3D53R7EA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.858549726 (318h54m39.898122856s) [noise=true , address=145.79.227.13:4803 , piece_id=KDBJVPCN23EKABNWWVVRX4AQMEJDQZARISJ5RJUHOUTDPEV7WDCQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:03.894730642 (318h54m39.934303762s) [noise=true , address=145.79.227.4:4185 , piece_id=FQHMOQX44FLUDFTJGUBX5DWDJFW6V6HSM4D63KXYEXYHV5AOET4A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: <nil> Sep 19 20:23:30.782372406 (318h55m6.821945536s) [noise=true , address=77.90.31.10:4520 , piece_id=4ZZTBBCXLUUOZX4NFZCKTFVSE5E2NPSLZGZVJMO454BGSGBIRKUQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: rpc: dial tcp 77.90.31.10:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782566216 (318h55m6.822139346s) [noise=true , address=45.91.116.14:4191 , piece_id=UKRQ3XKVA5T4GNMC7FX4KJHAX55M2PFKIN3GMTS55RBYKOPZCD2A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782738233 (318h55m6.822311363s) [noise=true , address=77.90.31.19:4843 , piece_id=ROQYWL4Z5ZWLABQEHZQFID4WV7KTELQA7J2OA75UKXA7JDZWNM3A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: read tcp 188.166.225.170:36868->77.90.31.19:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782802095 (318h55m6.822375225s) [noise=true , address=77.90.31.18:4807 , piece_id=574TE76APOATAV33FUWVTJUWCYTJS7YYLDFD46HRE2FZUVY7DM2Q, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: read tcp 188.166.225.170:49696->77.90.31.18:4807: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782857320 (318h55m6.82243046s) [noise=true , address=77.90.31.21:4879 , piece_id=272L36GDXMKFECALY2XBKPV3TMZ6PLK4ILPG5XWFACSDZ5Z3KA6A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:51488->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.782926932 (318h55m6.822500062s) [noise=true , address=45.91.116.4:4938 , piece_id=QGQ6LI5COBHXBJ4CLD5WEJ33D7NMTCXFPVYAG3ZMEFDQPTWPM24A, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: read tcp 188.166.225.170:58614->45.91.116.4:4938: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783033776 (318h55m6.822606926s) [noise=true , address=77.90.31.12:4589 , piece_id=ET6A5WH37QE2WYZZLXFOMULCJCSQFIHOGTOWTZIPTSNTC2ZD6HZQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: read tcp 188.166.225.170:49456->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783655770 (318h55m6.8232289s) [noise=true , address=45.91.116.13:4153 , piece_id=3NNOTDYJMWRJXUC3NHAJC22B2PRBQAWCFOW4CPXJLT3RTKJ5T3OA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, address:45.91.116.13:4153): protocol: rpc: dial tcp 45.91.116.13:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783779817 (318h55m6.823352947s) [noise=true , address=45.91.116.7:4010 , piece_id=XIBTQ7SN6KGPSXNEIYBK7ARYODOMAD2NB7VRARF2ENGHXZTYO4BA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, address:45.91.116.7:4010): protocol: read tcp 188.166.225.170:58732->45.91.116.7:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.783817048 (318h55m6.823390178s) [noise=true , address=45.91.116.14:4190 , piece_id=VERLABLD7AHLLKCHC2E7UUXQLW63ELBF25INW3YPQPQWCGSI6LQQ, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, address:45.91.116.14:4190): protocol: rpc: dial tcp 45.91.116.14:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784095428 (318h55m6.823668558s) [noise=true , address=45.91.116.9:4262 , piece_id=PTUPYBYQ4BH32JGHV6CRO72HEX7T2KBTTSZC7ZW6KN4I6OXXTATA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, address:45.91.116.9:4262): protocol: read tcp 188.166.225.170:52988->45.91.116.9:4262: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784174268 (318h55m6.823747398s) [noise=true , address=45.91.116.6:4973 , piece_id=OHQPQHU2DYS7PX7IERAZDHXTT5AT7DMHSRMYDDS4XIN2D6VPVFOA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, address:45.91.116.6:4973): protocol: read tcp 188.166.225.170:46200->45.91.116.6:4973: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784182323 (318h55m6.823755443s) [noise=true , address=45.91.116.11:4119 , piece_id=PTSE7XVAHZX4FIY7LGVXCDVS5JHLHPONJPWATSRAGLAXLHU6MXPA, seg_pos={0 0}, upload=280330]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: read tcp 188.166.225.170:40198->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Sep 19 20:23:30.784203453 (318h55m6.823776583s) [seg_pos={0 0}, upload=280330]: Done waiting for segment. successful: 3 optimal: 13 errs: <nil> Sep 19 20:23:31.029468838 (318h55m7.069041968s) [upload=280330]: Done uploading segments... Sep 19 20:23:31.029503895 (318h55m7.069077015s) [upload=280330]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 3) Duration: 6.070380228s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 6.251076803s FAIL (2025-09-19 20:47:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== 9 20:17:12.364393699 (318h48m48.403966819s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365023129 (318h48m48.404596259s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365397501 (318h48m48.404970631s) [seg_pos={0 0}, upload=280240]: Done begin upload segment. Sep 19 20:17:12.365425184 (318h48m48.404998304s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.365693655 (318h48m48.405266785s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.366524177 (318h48m48.406097307s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367034738 (318h48m48.406607868s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367515544 (318h48m48.407088664s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.367957696 (318h48m48.407530826s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.368606221 (318h48m48.408179351s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369057279 (318h48m48.40863041s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369432334 (318h48m48.409005454s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.369919091 (318h48m48.409492221s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370266703 (318h48m48.409839833s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.370632389 (318h48m48.41020551s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.371711514 (318h48m48.411284644s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.372313190 (318h48m48.41188632s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374011694 (318h48m48.413584824s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:12.374470608 (318h48m48.414043738s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Uploading piece... Sep 19 20:17:14.019248779 (318h48m50.058821909s) [upload=280240]: Next returned nil segment Sep 19 20:17:14.019270561 (318h48m50.058843691s) [upload=280240]: Waiting for error group managing segments... Sep 19 20:17:14.476629547 (318h48m50.516202677s) [noise=true , address=145.79.227.17:4976 , piece_id=MU6IWJJXJQ23AWUVSCSBSM4A7RB5PZ7STALZSMXYHMPUEHG56BJQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.489083742 (318h48m50.528656862s) [noise=true , address=145.79.227.11:4756 , piece_id=KG3BTQ67YTUVM7HX3MOTS4TJBJ4FHGIXSLO4R27JYNYSDX3J4NNA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.496168463 (318h48m50.535741593s) [noise=true , address=145.79.227.10:4426 , piece_id=MNKO7DAU55NU4E7XOAIH4E2KOFP7MOO4IGHQJOBX5GETLJVHWSIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.497793137 (318h48m50.537366267s) [noise=true , address=145.79.227.25:4620 , piece_id=6LQHZDFIXZNIVZGFPOOF4TBZHJNWXEGCBZC6KR7SSK7GBL4NL76A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.536117044 (318h48m50.575690164s) [noise=true , address=145.79.227.18:4010 , piece_id=VTYB3GRMFYRA6PIQA6HTNZ6OTDAI6HCINGPQT7Q4IC3DHL2S3SBQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.554736649 (318h48m50.594309779s) [noise=true , address=145.79.227.33:4887 , piece_id=5NEX26YZT2SFLJQ7TUZB7XATBEZLGKUXMLVSYDPC7AQ3GGWXRLIA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.564556408 (318h48m50.604129538s) [noise=true , address=145.79.227.11:4761 , piece_id=MNS6IL5H7VTFQC3GAWR7FHQUTNUOAKFKUQQYXWB7UH2KRU6O5MNQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.565922639 (318h48m50.605495769s) [noise=true , address=145.79.227.28:4757 , piece_id=PKHXGIK7EAUGAMHKHSWRXJSBEKJTGKO66RQFP6YDMFIVMYMI7X2A, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.633623536 (318h48m50.673196666s) [noise=true , address=145.79.227.43:4162 , piece_id=PEMG4INVP7ZAKXK5GVGFC63ULWB7IKD56ZUUY4ZF327HMM7OBYHA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.641636744 (318h48m50.681209874s) [noise=true , address=145.79.227.52:4466 , piece_id=X4OFH6OUQIZJQNJ2BGWRAYK72B4IEALJJVHWDT3NG7LPBPTZYFRA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:14.651220584 (318h48m50.690793714s) [noise=true , address=145.79.227.30:4763 , piece_id=FNOBK3PB3H2C44WZEHGYTKWTFUYU57L33KXIBRB4R3NPIKX2J7ZA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:17:17.959005140 (318h48m53.99857827s) [noise=true , address=145.79.227.19:4105 , piece_id=4NAR6IXHMB3YARSI35QWY4UTMWJPCK2MMKUBGAHQF3MEA2SB7Q3Q, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:10.524759042 (318h49m46.564332172s) [noise=true , address=145.79.227.40:4070 , piece_id=2SRJTCYR3JIIWPSNMBN4JQVDKMXMNXWKLZPEVNC4HXDYT6Z5V7LA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: <nil> Sep 19 20:18:11.204920837 (318h49m47.244493967s) [noise=true , address=145.79.227.24:4612 , piece_id=NJEA6IJWUONDL3SOAUOM435F7BAHIYJEOXYBQVDLKPSGBDGQOBZQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12jM3gpXbW8JvCB5bB2z3FoFmTEfZ9nsLeonzKzYmFT4SjyXUZV, address:145.79.227.24:4612): protocol: expected piece hash; context deadline exceeded; EOF Sep 19 20:18:11.204974739 (318h49m47.244547869s) [noise=true , address=145.79.227.46:4631 , piece_id=2PA2JR4CVLWOFUUXAOD7HEJFPX5L5GVAIJOVZWSRDRY2SXFLIEMA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12fNXZups4HA6xvjeWtLvpK2GfZYNFVkFZQ4e4L4sN11iwFZEvz, address:145.79.227.46:4631): protocol: expected piece hash; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; write tcp 188.166.225.170:45798->145.79.227.46:4631: use of closed network connection; EOF Sep 19 20:18:11.205059481 (318h49m47.244632601s) [noise=true , address=145.79.227.12:4776 , piece_id=KZQKVGORJL63HNSPDTP2PCA52YZGB5RCLRCVDLHDUWE7FKICE7WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:12AHK7d2cZxD8uFmiR7dyowGKW5YDnH3D2gVyEYxhjmaPScnxtk, address:145.79.227.12:4776): protocol: expected piece hash; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; write tcp 188.166.225.170:33952->145.79.227.12:4776: use of closed network connection; EOF Sep 19 20:18:11.205077004 (318h49m47.244650134s) [noise=true , address=145.79.227.14:4869 , piece_id=5GPUEZSW62BUWEXZPLRP2M6UCXHODDHIPXOXMFWMTBDWDFWBB6WA, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1273igSPBbnYnP5bTMPjCphwkCSK4WdBdBMFVgfnrPaq4oSUZQr, address:145.79.227.14:4869): protocol: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54162->145.79.227.14:4869: use of closed network connection Sep 19 20:18:11.205182405 (318h49m47.244755525s) [noise=true , address=145.79.227.5:4239 , piece_id=V7UMRJ2TBD3XPBV64YWSA7MJ7Y2B6EKWLYFLO3HXSGUFKT22U4VQ, seg_pos={0 0}, upload=280240]: Done uploading piece. err: ecclient: upload failed (node:1pTRYQDUQVz3nNo46TfMeaihJoP2zvTMfgvTNjUMF8iePz5MvV, address:145.79.227.5:4239): protocol: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36150->145.79.227.5:4239: use of closed network connection Sep 19 20:18:11.205211060 (318h49m47.24478419s) [seg_pos={0 0}, upload=280240]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Sep 19 20:18:11.446484774 (318h49m47.486057904s) [upload=280240]: Done uploading segments... Sep 19 20:18:11.446523338 (318h49m47.486096468s) [upload=280240]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 6.251110749s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)