Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [..............................................X.......X.XX..] duration: 5.858184883s FAIL (2025-07-28 00:22:35 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.858207606s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [...........................................X...XXXXXXXXXXXXX] duration: 1m0.235410925s FAIL (2025-07-28 00:21:57 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== CM4P7CSST75SNL56XJ7Q, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394239288 (84h45m8.116802328s) [noise=true , address=104.36.82.6:4879 , piece_id=RUYOAVTD77CHSLKSE7ZOQFBJJ4QBDJ3GWNCQTZDKSCXH4HXLD3DQ, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394442326 (84h45m8.117005367s) [noise=true , address=104.36.82.55:4202 , piece_id=WJE62QI7YVUGLWV7UMRNUDH5GZPZY6O44MKRWUBZV2KBYHE7M4MA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394642330 (84h45m8.11720537s) [noise=true , address=104.36.82.52:4104 , piece_id=GIDJCXWDKL33QIQNYMV4RSXXERZNNX5ZM2STNNT74RBKLOAY5GKA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.394852391 (84h45m8.117415431s) [noise=true , address=104.36.82.9:4014 , piece_id=WPGIYXAVRE5LNAC7BTIMYM2ZLSFK4I7EKBFNUZHXCQXFW2CKIK7Q, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:20:58.395072282 (84h45m8.117635322s) [noise=true , address=104.36.82.53:4130 , piece_id=JFZZSYH52WQ6EBH6ZPKR7XPU22PXD3FMPX4TQPKU3YN77FL3SHTA, seg_pos={0 0}, upload=81337]: Uploading piece... Jul 28 00:21:57.287344145 (84h46m7.009907195s) [upload=81337]: Next segment err: context deadline exceeded Jul 28 00:21:57.287832549 (84h46m7.01039559s) [noise=true , address=104.36.82.49:4985 , piece_id=QJYTPFDOMOQ5SQCQV5XJEQ4SHRYAUV64WSVFLH5JCLRMLOMC2LUA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1rp73cCs69rhnqNr1dPffoWvGNezRvcpC5WAbUbaNbkbuCfUsJ, address:104.36.82.49:4985): protocol: write tcp 188.166.225.170:55326->104.36.82.49:4985: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287873427 (84h46m7.010436467s) [noise=true , address=104.36.82.46:4498 , piece_id=XTAL5NZOVEUKKQ5AEVVFCZNGB3T5DSQOEP2ZFZ3C464PE2BWRNMQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: write tcp 188.166.225.170:54314->104.36.82.46:4498: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287907683 (84h46m7.010470724s) [noise=true , address=104.36.82.6:4879 , piece_id=RUYOAVTD77CHSLKSE7ZOQFBJJ4QBDJ3GWNCQTZDKSCXH4HXLD3DQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12jH7W99nQ77T67b8WVVqmg9Jqi5QoB1qM5Wcj8WexywAZ7fRNT, address:104.36.82.6:4879): protocol: write tcp 188.166.225.170:46190->104.36.82.6:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.287945856 (84h46m7.010508906s) [noise=true , address=104.36.82.53:4130 , piece_id=JFZZSYH52WQ6EBH6ZPKR7XPU22PXD3FMPX4TQPKU3YN77FL3SHTA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12N8MGboFyEV381rkrBva3G3advz3QQEY2HCHxwqzKX2X5zxNvb, address:104.36.82.53:4130): protocol: write tcp 188.166.225.170:48636->104.36.82.53:4130: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288059634 (84h46m7.010622674s) [noise=true , address=104.36.82.42:4798 , piece_id=RXFPR7QOYXHUKDUDSGKDVYG2YR4HV7XZ2PGDNFJGWHG6C3RPZLWQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:104.36.82.42:4798): protocol: write tcp 188.166.225.170:51402->104.36.82.42:4798: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288121732 (84h46m7.010684772s) [noise=true , address=104.36.82.50:4043 , piece_id=BZYCXUKWSWJV47EZRGTXFZYNMKPSBMHAYTKW2LC767M7H6NKU7YQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12f97TAyWcE4YrxDQCeycC8tnuetQiagu2MV4U5o8ARLq8jGKTX, address:104.36.82.50:4043): protocol: write tcp 188.166.225.170:58482->104.36.82.50:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288159245 (84h46m7.010722285s) [noise=true , address=104.36.82.36:4632 , piece_id=U276W6EIX52I3CG3BJ3VV7ANMPUYIFBQNIOCEUT4XR7IG3UMW2TA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1B7uLDPBrYvc1N81dKm8bcYnhYwjA5mRdRfeQpWqKqrJjmQJgC, address:104.36.82.36:4632): protocol: write tcp 188.166.225.170:40544->104.36.82.36:4632: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288203669 (84h46m7.010766709s) [noise=true , address=104.36.82.10:4090 , piece_id=RRPNRXMQGBFWVSL6IJP6C5EHDUHQVUUHCM4P7CSST75SNL56XJ7Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:104.36.82.10:4090): protocol: write tcp 188.166.225.170:60812->104.36.82.10:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288239388 (84h46m7.010802428s) [noise=true , address=104.36.82.52:4104 , piece_id=GIDJCXWDKL33QIQNYMV4RSXXERZNNX5ZM2STNNT74RBKLOAY5GKA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12qmPpCmXAvuJZn2KR5Kr4nHYwKEzWvwqZ82nqNka5u1mKaZmXs, address:104.36.82.52:4104): protocol: write tcp 188.166.225.170:38968->104.36.82.52:4104: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288277020 (84h46m7.01084006s) [noise=true , address=104.36.82.54:4175 , piece_id=2WV4ISHIQZ3NQXIUG5PRMWXV2NGUZ5Q5VKGZQ2KUIPT7YI2OOSHA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: write tcp 188.166.225.170:57302->104.36.82.54:4175: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288441444 (84h46m7.011004484s) [noise=true , address=104.36.82.14:4242 , piece_id=RXXG7CX2O4C4333X7EHQOUHR4ZGPLBA7K66PJYOEMO5W4W6IZVWQ, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 188.166.225.170:60670->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288512010 (84h46m7.01107506s) [noise=true , address=104.36.82.56:4258 , piece_id=WGLDNQYGXJKCDXYSRM72H7GECMRHECFFEPZ25IJQLMWJQGTBSE4A, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:35042->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288578076 (84h46m7.011141156s) [noise=true , address=104.36.82.12:4174 , piece_id=2QAW6BIB4I74IKMBZEDN5A2PV7GZBMKPL4IMYWDZUGBWVWFY454A, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ZqPqvh4SVpDmkpVKQXZUW48CtzV9VMeMGJoDZwA7KpNfmjfow, address:104.36.82.12:4174): protocol: write tcp 188.166.225.170:37372->104.36.82.12:4174: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288669540 (84h46m7.011232611s) [noise=true , address=104.36.82.9:4014 , piece_id=WPGIYXAVRE5LNAC7BTIMYM2ZLSFK4I7EKBFNUZHXCQXFW2CKIK7Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12uK9ULUdABDm5XBP4nh1omWhR5qoQerFsCmL4bQ9dmNespXpTy, address:104.36.82.9:4014): protocol: write tcp 188.166.225.170:32806->104.36.82.9:4014: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288696572 (84h46m7.011259612s) [noise=true , address=104.36.82.7:4942 , piece_id=HGXN6AR6HVLTCDTRO6Q7UBSZDQ7JY36M52U7OK5M5RR5TH43EN3Q, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12ArPftAghEJaoHeAEPqGK9QDhRguh3yEovFUjkEMSDBdsfAVnC, address:104.36.82.7:4942): protocol: write tcp 188.166.225.170:52758->104.36.82.7:4942: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288748462 (84h46m7.011311502s) [noise=true , address=104.36.82.55:4202 , piece_id=WJE62QI7YVUGLWV7UMRNUDH5GZPZY6O44MKRWUBZV2KBYHE7M4MA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: write tcp 188.166.225.170:58294->104.36.82.55:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288755485 (84h46m7.011318525s) [noise=true , address=104.36.82.15:4280 , piece_id=XSWP44ZZGA2JCQA6LQVCM3TIYNFGKJE7X427Q7ABVKSBZUKRT2EA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:1t4hMpMGCu1YMmNLX6fssTjWy9Xi5Fp8kdHTDmkhxH9qsZiYM6, address:104.36.82.15:4280): protocol: write tcp 188.166.225.170:60296->104.36.82.15:4280: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288784791 (84h46m7.011347831s) [noise=true , address=104.36.82.11:4125 , piece_id=GHLIH2KBA6WEA53ZXLJ3NTQRGY46OHN4R5ZG365VAIKPRZMRHKZA, seg_pos={0 0}, upload=81337]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: write tcp 188.166.225.170:43330->104.36.82.11:4125: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:21:57.288825339 (84h46m7.011388379s) [seg_pos={0 0}, upload=81337]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.235422326s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X............X...............] duration: 1.920898625s FAIL (2025-07-28 00:21:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.920954442s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 5.841853422s FAIL (2025-07-28 00:21:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841882569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............................................X.......X.XX.] duration: 5.820141665s FAIL (2025-07-28 00:21:32 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 5.820169569s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X............X..............] duration: 1.845903813s FAIL (2025-07-28 00:21:31 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.845932357s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 5.824791367s FAIL (2025-07-28 00:21:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.824824901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X............X.............] duration: 2.133415321s FAIL (2025-07-28 00:21:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.133444076s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [............................................X...XXXXXXXXXXXX] duration: 1m0.239743358s FAIL (2025-07-28 00:20:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== Jul 28 00:19:57.906994140 (84h44m7.62955718s) [noise=true , address=104.36.82.11:4137 , piece_id=YQC4N2XEZZR6Q4XXMS3NUW7GLHF2JVQP6U2IH3XPEFPNJD23AMVQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907128808 (84h44m7.629691838s) [noise=true , address=104.36.82.4:4831 , piece_id=RGGSZZVZXNG4AYAKAVJ4NCZF233LYIIEQIQZ77RHMUUJ67XF6CFA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907285447 (84h44m7.629848477s) [noise=true , address=104.36.82.14:4253 , piece_id=6D5W2GS5UI4EN5ODH446BZILZJIE5NFDVAHBGXVLJR2DOUJ3D4DA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907359248 (84h44m7.629922278s) [noise=true , address=104.36.82.53:4150 , piece_id=NH2SET23UMC75UWQ7LN3O4JCQYEENWEE7UXLJLPFOZWVCXJEXQZQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907579319 (84h44m7.630142359s) [noise=true , address=104.36.82.6:4897 , piece_id=ZDQBPT3KIMDFS44JL3YOJVVKYRXIK56IGBGJ35OPJGPYVXJVIU3A, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907681365 (84h44m7.630244405s) [noise=true , address=79.127.201.215:4360 , piece_id=YXCCPQ3TFYRMJFL3ESWPEG73WNEM642BHBMBUJY47DU7WT2VXJTQ, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.907895695 (84h44m7.630458735s) [noise=true , address=104.36.82.54:4173 , piece_id=LKPF6JIDWQOYZLMZURODZLFTNTODGHDC2KPKKRYPACWEDBYL6U7Q, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:57.908148118 (84h44m7.630711148s) [noise=true , address=104.36.82.49:4002 , piece_id=WUCF4NTRVRJDJQU6AXHEMSVWKJN4N7SP5EL6NLD7P3U6OEXMESRA, seg_pos={0 0}, upload=81322]: Uploading piece... Jul 28 00:19:59.417184952 (84h44m9.139747992s) [upload=81322]: Next returned nil segment Jul 28 00:19:59.417201624 (84h44m9.139764664s) [upload=81322]: Waiting for error group managing segments... Jul 28 00:20:01.618731737 (84h44m11.341294777s) [noise=true , address=79.127.201.215:4360 , piece_id=YXCCPQ3TFYRMJFL3ESWPEG73WNEM642BHBMBUJY47DU7WT2VXJTQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: <nil> Jul 28 00:20:56.806154697 (84h45m6.528717747s) [noise=true , address=104.36.82.11:4137 , piece_id=YQC4N2XEZZR6Q4XXMS3NUW7GLHF2JVQP6U2IH3XPEFPNJD23AMVQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:104.36.82.11:4137): protocol: rpc: dial tcp 104.36.82.11:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806261239 (84h45m6.52882428s) [noise=true , address=104.36.82.5:4847 , piece_id=7DX6JDL2RUQONZBBVBAGINWXM3Y45ZPZNYKP3YDCFDDH63ME4BUA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: rpc: dial tcp 104.36.82.5:4847: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806371210 (84h45m6.52893426s) [noise=true , address=104.36.82.41:4771 , piece_id=ERGGVKVMJYT2FOLVYIBJLMNTBOAWKGEP7T7NK67S2JO3DEYDTRPA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806428981 (84h45m6.528992021s) [noise=true , address=104.36.82.9:4012 , piece_id=HR5NRTEQOVO2J446PWCX5XTTPCRO4Z5PRXDATEF5PZYTXBEE5QPQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:15gHkND99Ksp8TdyAbAvMUymRoLDyqvbdcy9Y3Hjhpb8zTpCHi, address:104.36.82.9:4012): protocol: rpc: dial tcp 104.36.82.9:4012: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806463738 (84h45m6.529026778s) [noise=true , address=104.36.82.46:4508 , piece_id=HRNWSQC5OUUWKF2IE5EVGEPMVWDJSTGSW6AC4MZYSD3RUPTK6SAQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806502431 (84h45m6.529065471s) [noise=true , address=104.36.82.51:4053 , piece_id=4MA3SWYEIZGJCVAXRPYV2ZMF3RGVJPPCXD2WBRILJZBKJ7IW3QZA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12DqyTc4MHvxutFXs2WWRCk8MSuxF3Y1HayyQY4Y5sy8cqDLWuG, address:104.36.82.51:4053): protocol: rpc: dial tcp 104.36.82.51:4053: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806538690 (84h45m6.52910173s) [noise=true , address=104.36.82.4:4831 , piece_id=RGGSZZVZXNG4AYAKAVJ4NCZF233LYIIEQIQZ77RHMUUJ67XF6CFA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1251oUFjESKjnTubb2wF8K9nsa8fztBm4nup3GvAymHLExfH35v, address:104.36.82.4:4831): protocol: rpc: dial tcp 104.36.82.4:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806573928 (84h45m6.529136968s) [noise=true , address=104.36.82.14:4253 , piece_id=6D5W2GS5UI4EN5ODH446BZILZJIE5NFDVAHBGXVLJR2DOUJ3D4DA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806611400 (84h45m6.52917444s) [noise=true , address=104.36.82.20:4456 , piece_id=72LVGO6GBBUH3ZUTH5R5ZXDZUV6CGI4YN6MG7DT3D4ZPMDFDQC4A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1TmBh7KCuUydq7ZnVtJR4cWKa6ktWH4me19r8vGdggGxohDUyv, address:104.36.82.20:4456): protocol: write tcp 188.166.225.170:36146->104.36.82.20:4456: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806809559 (84h45m6.529372599s) [noise=true , address=104.36.82.15:4275 , piece_id=NG3ISLHLF5M2FZYQQVDK6523NBL7CFFQG7KSQIFUE4ADNJL3CULQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12EzSYYm7aKxspeyC4hZCxZzCdcSNMyDTJx3TYcci5bVT6rZLbr, address:104.36.82.15:4275): protocol: rpc: dial tcp 104.36.82.15:4275: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.806923376 (84h45m6.529486426s) [noise=true , address=104.36.82.10:4096 , piece_id=ZMMS7B35OUXL6XPEUCZUTI4QT3FOODKW5M7FN33SQS26QAHT6S2A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: rpc: dial tcp 104.36.82.10:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807009181 (84h45m6.529572221s) [noise=true , address=104.36.82.6:4897 , piece_id=ZDQBPT3KIMDFS44JL3YOJVVKYRXIK56IGBGJ35OPJGPYVXJVIU3A, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12qtpRnAcGiHmdR6J6ysdwaLPvejnCvpKm6MggqsezhbZ14Z5Bt, address:104.36.82.6:4897): protocol: rpc: dial tcp 104.36.82.6:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807015794 (84h45m6.529578834s) [noise=true , address=104.36.82.56:4255 , piece_id=VEVSTLO4GUHKIIRVG64D2U52ANQPHVZ4M2MUKFLQICAIZ2J34UCA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12S6rwimXBvaEb4KneAwL5H3t5zTmvGVhyJSWCTU1u17r4RWJta, address:104.36.82.56:4255): protocol: rpc: dial tcp 104.36.82.56:4255: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807050169 (84h45m6.529613209s) [noise=true , address=104.36.82.49:4002 , piece_id=WUCF4NTRVRJDJQU6AXHEMSVWKJN4N7SP5EL6NLD7P3U6OEXMESRA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12FgCn3deV8CMPtLrNN1oVxTwRE8c5RrsKEs9bna5MntLDM9Dzo, address:104.36.82.49:4002): protocol: rpc: dial tcp 104.36.82.49:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807088282 (84h45m6.529651322s) [noise=true , address=104.36.82.53:4133 , piece_id=5PDEOLN4QJB4RS6ANJIWOCALNZZQDOAF23QV2ELJFRY6ZS4AJ5YA, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807101417 (84h45m6.529664478s) [noise=true , address=104.36.82.53:4150 , piece_id=NH2SET23UMC75UWQ7LN3O4JCQYEENWEE7UXLJLPFOZWVCXJEXQZQ, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1jDz53H9KVPn5dBA1gqj66ZQEKbo9dNLucS9c3u8WvY1Xn7jH, address:104.36.82.53:4150): protocol: rpc: dial tcp 104.36.82.53:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807131305 (84h45m6.529694345s) [noise=true , address=104.36.82.54:4173 , piece_id=LKPF6JIDWQOYZLMZURODZLFTNTODGHDC2KPKKRYPACWEDBYL6U7Q, seg_pos={0 0}, upload=81322]: Done uploading piece. err: ecclient: upload failed (node:1w2v4iyYtJtcdz7shYhG8CjCDQ1dWjeJVwY5Ubi7Ex5Y38e2i1, address:104.36.82.54:4173): protocol: write tcp 188.166.225.170:44964->104.36.82.54:4173: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:20:56.807167974 (84h45m6.529731034s) [seg_pos={0 0}, upload=81322]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:20:57.043655985 (84h45m6.766219025s) [upload=81322]: Done uploading segments... Jul 28 00:20:57.043682235 (84h45m6.766245275s) [upload=81322]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.23975456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X............X............] duration: 1.902202206s FAIL (2025-07-28 00:20:37 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.902225211s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [................................................X.......X.XX] duration: 4.075580564s FAIL (2025-07-28 00:20:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 4.075608299s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 5.780575958s FAIL (2025-07-28 00:20:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.780602227s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X............X...........] duration: 1.860872301s FAIL (2025-07-28 00:19:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.860895195s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [.............................................X...XXXXXXXXXXX] duration: 1m0.238740178s FAIL (2025-07-28 00:19:56 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== N6YU47IQ2RA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.440410160 (84h43m7.16297321s) [noise=true , address=104.36.82.40:4973 , piece_id=UWOSZID6YLTHO54RBS4ZK3SFK2PA4OVTJB2VYY655KMFTTTAZ4VQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.440737477 (84h43m7.163300527s) [noise=true , address=104.36.82.18:4396 , piece_id=RWV6DS3ELZM5IAT2WX22T25UWIYMBRRJAUKMA4QSGN45PO3MRQAA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441031469 (84h43m7.163594509s) [noise=true , address=104.36.82.56:4246 , piece_id=NHHZS7SADGQTZSWUEM6PUYT7LMXKGSLIAEO4YZDZXXZTENWCVFFA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441321363 (84h43m7.163884414s) [noise=true , address=104.36.82.16:4301 , piece_id=VWKOS6JM7SSSSGRUYHJDVYQJLT645Z4NJYRF3TU4VXBVHSDID4OA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.441595598 (84h43m7.164158628s) [noise=true , address=104.36.82.55:4202 , piece_id=QAPD6CCJV5V5O5S2O5JYHSD6O5YU7DIKP3NSXL3FY4OPHLDK35XA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442018017 (84h43m7.164581057s) [noise=true , address=104.36.82.53:4154 , piece_id=OQROWXS6GFWUNVE43TS6F3QMFXLIKSHEARAAYAV2WZJMPMMJJTQQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442098872 (84h43m7.164661912s) [noise=true , address=104.36.82.42:4807 , piece_id=MXI7PPAP6PXKQGO67NESW6ZVWFLF4HRENZKRGXOTA3JF7LTCIQJQ, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:18:57.442410307 (84h43m7.164973347s) [noise=true , address=104.36.82.50:4032 , piece_id=DSXLYFSLOZY2PB73IYN2TDJWIXUGXMEPHDUWOGHJ3QR3FWV75UBA, seg_pos={0 0}, upload=81307]: Uploading piece... Jul 28 00:19:56.344684181 (84h44m6.067247221s) [upload=81307]: Next segment err: context deadline exceeded Jul 28 00:19:56.345169609 (84h44m6.06773265s) [noise=true , address=104.36.82.21:4507 , piece_id=RYD5CA2MZF4ZG5VL4SG5LDKAN2VQ2RSGKXRXOD275HK4IO2V3QOQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345309437 (84h44m6.067872477s) [noise=true , address=104.36.82.55:4202 , piece_id=QAPD6CCJV5V5O5S2O5JYHSD6O5YU7DIKP3NSXL3FY4OPHLDK35XA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345347060 (84h44m6.06791009s) [noise=true , address=104.36.82.42:4807 , piece_id=MXI7PPAP6PXKQGO67NESW6ZVWFLF4HRENZKRGXOTA3JF7LTCIQJQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345376866 (84h44m6.067939897s) [noise=true , address=104.36.82.54:4190 , piece_id=ZH5TFAYYBR2NDUFBKRZPQBBIFGXR6VBAIJKMNUYCF5TARZIGAGOA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1eKHcxWuBbQAGX44kVrn9cfEhEcRpmQfQegcoMGjr92pfM1VVF, address:104.36.82.54:4190): protocol: rpc: dial tcp 104.36.82.54:4190: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345408647 (84h44m6.067971687s) [noise=true , address=104.36.82.12:4167 , piece_id=6NCVVGGNFC3TZQAZRQ5XGSND3KKJWE5SJRUDTTF53MOSUYUAXZ3Q, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: write tcp 188.166.225.170:59842->104.36.82.12:4167: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.345436170 (84h44m6.06799921s) [noise=true , address=104.36.82.18:4396 , piece_id=RWV6DS3ELZM5IAT2WX22T25UWIYMBRRJAUKMA4QSGN45PO3MRQAA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1k5gzJBGWhpARcfiynburgfTqJ2t7sVNs1mAkUfToMG5saxPqx, address:104.36.82.18:4396): protocol: rpc: dial tcp 104.36.82.18:4396: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346098667 (84h44m6.068661707s) [noise=true , address=104.36.82.52:4112 , piece_id=BDBGZ5LXAWLK6BKOXQDLH3QVGP3MGGSYAHKGZEBUVJGXPL5ZARRQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1dQFyMxQ1vVeHWCmyUWFUtJNBTdRUP2bbdLJQCMwF9z1L1eapD, address:104.36.82.52:4112): protocol: rpc: dial tcp 104.36.82.52:4112: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346133043 (84h44m6.068696083s) [noise=true , address=104.36.82.9:4009 , piece_id=QHGJVTXXRGOBNYKRJ2WXXAZ3MSGTLX2WDGEAUC3GMPSMHMUMHUIQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346161317 (84h44m6.068724357s) [noise=true , address=104.36.82.16:4301 , piece_id=VWKOS6JM7SSSSGRUYHJDVYQJLT645Z4NJYRF3TU4VXBVHSDID4OA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: write tcp 188.166.225.170:47256->104.36.82.16:4301: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346187537 (84h44m6.068750577s) [noise=true , address=104.36.82.15:4285 , piece_id=NR246ARALGUBMGOUG7JAI4HP67PJ76RFWXDH2HFLAU4U2TBTUWMQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:18UBLXtDVgwsDAPguXZ7eWhE9XKwfa8US7HXbWaXuhpDzCLa8Y, address:104.36.82.15:4285): protocol: rpc: dial tcp 104.36.82.15:4285: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346216452 (84h44m6.068779492s) [noise=true , address=104.36.82.53:4154 , piece_id=OQROWXS6GFWUNVE43TS6F3QMFXLIKSHEARAAYAV2WZJMPMMJJTQQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346243514 (84h44m6.068806574s) [noise=true , address=104.36.82.14:4256 , piece_id=AHYSCPLF24PX2GNEWXHE22WD5D7NY2OLMFWJGSO4VQJOKKWUUWVQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12rxUCiyihymapVWXRYTgyrg2VMHxMe1HnYiV9JCKgaRyiFLiPm, address:104.36.82.14:4256): protocol: rpc: dial tcp 104.36.82.14:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346274313 (84h44m6.068837353s) [noise=true , address=104.36.82.13:4220 , piece_id=QZYTN33PLPI5CBLOU6PBTXOYRTCOULZNQ5OCDQCJUO5WI5QO6EXQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1AjQmw3H58hTwr2389cmMcwy1gBavBrZB8AWB1gqfoFRUkCpH6, address:104.36.82.13:4220): protocol: write tcp 188.166.225.170:33164->104.36.82.13:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346321614 (84h44m6.068884654s) [noise=true , address=104.36.82.40:4973 , piece_id=UWOSZID6YLTHO54RBS4ZK3SFK2PA4OVTJB2VYY655KMFTTTAZ4VQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1dCvsMCm5ZRmWVqP1zr9noHVuPeyrXHjpRvdHnKoAZqqt6knx, address:104.36.82.40:4973): protocol: rpc: dial tcp 104.36.82.40:4973: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346328537 (84h44m6.068891577s) [noise=true , address=104.36.82.47:4531 , piece_id=4W3REBM7FHQXX5X4ITFFMZJ3X4LTMLY3CLZ7GAVZKN6YU47IQ2RA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346336002 (84h44m6.068899042s) [noise=true , address=104.36.82.11:4133 , piece_id=RVWCVPHJZ6JBTXH2N622GBCF3QH2KK6OFR2CE3GXWUE34ZVMR3PQ, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346343215 (84h44m6.068906255s) [noise=true , address=104.36.82.50:4032 , piece_id=DSXLYFSLOZY2PB73IYN2TDJWIXUGXMEPHDUWOGHJ3QR3FWV75UBA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1zj1d7Yzk4Z5Wda3YF6UgR5oSEhABcf6HCWwmHWXaX1CbKYJdQ, address:104.36.82.50:4032): protocol: rpc: dial tcp 104.36.82.50:4032: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346358955 (84h44m6.068921985s) [noise=true , address=104.36.82.56:4246 , piece_id=NHHZS7SADGQTZSWUEM6PUYT7LMXKGSLIAEO4YZDZXXZTENWCVFFA, seg_pos={0 0}, upload=81307]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: rpc: dial tcp 104.36.82.56:4246: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:19:56.346391077 (84h44m6.068954107s) [seg_pos={0 0}, upload=81307]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.238757793s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 6.018631234s FAIL (2025-07-28 00:19:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.018651783s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.......X.X] duration: 30.227579459s FAIL (2025-07-28 00:19:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.227604787s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X............X..........] duration: 1.896335217s FAIL (2025-07-28 00:19:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.896362468s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..............................................X...XXXXXXXXXX] duration: 1m0.232186909s FAIL (2025-07-28 00:18:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== upload=81293]: Uploading piece... Jul 28 00:17:56.978157511 (84h42m6.700720551s) [noise=true , address=104.36.82.14:4224 , piece_id=2QUBF66SS3CDQZKT6NM6A6JKT7K3VQKA2JFF4ZV5SHFNMRCSPEOQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978329099 (84h42m6.700892129s) [noise=true , address=104.36.82.52:4115 , piece_id=H256VQY4J2E5XMLI2RJTWIATGKZFENZPMZ2YFIUJQQIK5YH5YFNQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978403171 (84h42m6.700966211s) [noise=true , address=104.36.82.16:4301 , piece_id=5IM4S3VGTBLDF6Y64OPFBZSZ3DVMGYQTOHQ4BYAL3OOFBKHIXMAA, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978629644 (84h42m6.701192684s) [noise=true , address=104.36.82.55:4222 , piece_id=A256DNPSGZBEOB3DNJW6PUAHAHGPWYH2HODCML5SSERCC4TTFUJQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978638331 (84h42m6.701201371s) [noise=true , address=104.36.82.12:4162 , piece_id=NXWDQ6LICRYIQ35NKFOM6JJC6PWAOUI532M72Y3GOWMQJLNIM74A, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978847832 (84h42m6.701410862s) [noise=true , address=104.36.82.41:4789 , piece_id=MZHEOD2O2BJHXXQN6FI37BCVOASM63LEVNGN5VHTTMQBVNSNRECQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.978891915 (84h42m6.701454956s) [noise=true , address=104.36.82.44:4900 , piece_id=I7MGHZEOVXZQZ7RKSRXLGDKLU24Z43X7OZXX2VWR2AIOOFXF7USQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979075247 (84h42m6.701638287s) [noise=true , address=79.127.220.101:4045 , piece_id=SXZUOR3GCKVPFYSKGXZC5DWB3PIMQPNJHEJHVIHSYZ4MSMPTUT7Q, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979124190 (84h42m6.70168723s) [noise=true , address=104.36.82.10:4097 , piece_id=4GMFJ5URDVSABJREK3QXGGWWFVXL6WB67V726OPSF66WKGZFABSQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:56.979372385 (84h42m6.701935425s) [noise=true , address=79.127.201.215:4343 , piece_id=3PDQ3HFYAUEPDPMQ3N4QZ74TYSSCZZUUTV4D6JPO5YYKLW6JZREQ, seg_pos={0 0}, upload=81293]: Uploading piece... Jul 28 00:17:58.073822806 (84h42m7.796385846s) [upload=81293]: Next returned nil segment Jul 28 00:17:58.073840519 (84h42m7.796403559s) [upload=81293]: Waiting for error group managing segments... Jul 28 00:17:58.358173788 (84h42m8.080736828s) [noise=true , address=79.127.205.235:4688 , piece_id=JGTIODKIDVXL6ZKRNKTBNS57YUUTRHM7KPM2M6G7WN2VFTGDXREQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:17:58.891902972 (84h42m8.614466012s) [noise=true , address=79.127.220.101:4045 , piece_id=SXZUOR3GCKVPFYSKGXZC5DWB3PIMQPNJHEJHVIHSYZ4MSMPTUT7Q, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:18:02.011626270 (84h42m11.73418931s) [noise=true , address=79.127.201.215:4343 , piece_id=3PDQ3HFYAUEPDPMQ3N4QZ74TYSSCZZUUTV4D6JPO5YYKLW6JZREQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: <nil> Jul 28 00:18:55.890590539 (84h43m5.613153569s) [noise=true , address=104.36.82.16:4301 , piece_id=5IM4S3VGTBLDF6Y64OPFBZSZ3DVMGYQTOHQ4BYAL3OOFBKHIXMAA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1GWFshsj2bhjcCVMnFSkiFN4qftcX9bXzG8h6VkZoHxhWcGRSG, address:104.36.82.16:4301): protocol: rpc: dial tcp 104.36.82.16:4301: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890635736 (84h43m5.613198766s) [noise=true , address=104.36.82.6:4898 , piece_id=PYH4GESFIJLJGM3SUF3CZKXFKD52K7TYWKIIVDHYJQ5XT5A255WA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1D4E49RAbRK64c95rK6tGHPHHo5xLsmZ53ZK9i2iRU8mz7SbZ8, address:104.36.82.6:4898): protocol: rpc: dial tcp 104.36.82.6:4898: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890667326 (84h43m5.613230366s) [noise=true , address=104.36.82.13:4213 , piece_id=4E7EVRG7XPF5D35VLKHFLVA3MO6F63OC5N6NI34QYS6XI2DJZQ4A, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12XDcAySWNCL7XoStP5GKzAksN1kTs9L8u4KStgyaiQn3AvLcRM, address:104.36.82.13:4213): protocol: rpc: dial tcp 104.36.82.13:4213: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890698335 (84h43m5.613261375s) [noise=true , address=104.36.82.44:4900 , piece_id=I7MGHZEOVXZQZ7RKSRXLGDKLU24Z43X7OZXX2VWR2AIOOFXF7USQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:104.36.82.44:4900): protocol: rpc: dial tcp 104.36.82.44:4900: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890728403 (84h43m5.613291443s) [noise=true , address=104.36.82.21:4507 , piece_id=FGZJDIKOHLMO2RWCMC5NJLMWRX5BP2RGYIEHO37LPSDVJQQMWXAA, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1XW3SaTU4fPuKGNeLtt4H9wQXw5ADEcvzcpo7J53wUULsMT5Bv, address:104.36.82.21:4507): protocol: rpc: dial tcp 104.36.82.21:4507: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890758611 (84h43m5.613321641s) [noise=true , address=104.36.82.55:4222 , piece_id=A256DNPSGZBEOB3DNJW6PUAHAHGPWYH2HODCML5SSERCC4TTFUJQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12ZKW35KvrsoYRefbYeUFZo8bh4XZYycNKGEvTFjzibezjXgj3e, address:104.36.82.55:4222): protocol: write tcp 188.166.225.170:59390->104.36.82.55:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890783869 (84h43m5.613346909s) [noise=true , address=104.36.82.52:4115 , piece_id=H256VQY4J2E5XMLI2RJTWIATGKZFENZPMZ2YFIUJQQIK5YH5YFNQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890802234 (84h43m5.613365274s) [noise=true , address=104.36.82.54:4191 , piece_id=GMZKUFLZZ42LW77H3E4GLBX5IHAUBM75OGQ73NCDBI2PD3LAIAQQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890832552 (84h43m5.613395592s) [noise=true , address=104.36.82.53:4146 , piece_id=OVYPVDGILZG52JCJ35HADZ7KR3TOGDTTVR3NGAH4BRNVNRNKPSIQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 188.166.225.170:47596->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890848122 (84h43m5.613411162s) [noise=true , address=104.36.82.14:4224 , piece_id=2QUBF66SS3CDQZKT6NM6A6JKT7K3VQKA2JFF4ZV5SHFNMRCSPEOQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: rpc: dial tcp 104.36.82.14:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890860345 (84h43m5.613423386s) [noise=true , address=104.36.82.41:4789 , piece_id=MZHEOD2O2BJHXXQN6FI37BCVOASM63LEVNGN5VHTTMQBVNSNRECQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1AXrW3TejqoDouALeZ1ddt8NqHZ5NZgZk5piqZHQNWgRjPHNqN, address:104.36.82.41:4789): protocol: rpc: dial tcp 104.36.82.41:4789: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890880544 (84h43m5.613443584s) [noise=true , address=104.36.82.12:4162 , piece_id=NXWDQ6LICRYIQ35NKFOM6JJC6PWAOUI532M72Y3GOWMQJLNIM74A, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1yxXQkfWVGwBuptpPjFgjhfTraAmE3WqtS3WEDbhQsJftjeaxV, address:104.36.82.12:4162): protocol: rpc: dial tcp 104.36.82.12:4162: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890892126 (84h43m5.613455166s) [noise=true , address=104.36.82.5:4855 , piece_id=IW7RZRWZ2GN4FQKWYHAXDNKR2RSP5BMNHWQXCU74MCRG35FYQE7Q, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1KDNuE7YY8smhDjotushQvY3y9NtLxigbJELHD4Edxy8Qi4BWh, address:104.36.82.5:4855): protocol: rpc: dial tcp 104.36.82.5:4855: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890922534 (84h43m5.613485574s) [noise=true , address=104.36.82.10:4097 , piece_id=4GMFJ5URDVSABJREK3QXGGWWFVXL6WB67V726OPSF66WKGZFABSQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:12QP2aZ9BC22cD1Nzz2E12DiA1XoNu3GGTgHC8Cc3TvpQM3Ubn3, address:104.36.82.10:4097): protocol: rpc: dial tcp 104.36.82.10:4097: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890931932 (84h43m5.613494962s) [noise=true , address=104.36.82.54:4169 , piece_id=HVCSJG4RPRBXSTKFZWYUDTSV7UAQQJUWLK65TNG44YZPDHINK3FQ, seg_pos={0 0}, upload=81293]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:18:55.890948454 (84h43m5.613511494s) [seg_pos={0 0}, upload=81293]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:18:56.120951736 (84h43m5.843514776s) [upload=81293]: Done uploading segments... Jul 28 00:18:56.120976824 (84h43m5.843539864s) [upload=81293]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.232201046s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 5.871233103s FAIL (2025-07-28 00:18:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871264923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.......X.] duration: 5.85053817s FAIL (2025-07-28 00:18:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.850560474s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 5.793558166s FAIL (2025-07-28 00:18:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793579047s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X............X.........] duration: 2.035964225s FAIL (2025-07-28 00:18:10 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.035991876s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.......X] duration: 30.230885213s FAIL (2025-07-28 00:18:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.230915011s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [...............................................X...XXXXXXXXX] duration: 1m0.234901758s FAIL (2025-07-28 00:17:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ce_id=V5DESVGF3MZF3Z7PSKJTLGANBLEYMQQMNQQRUZFUFBEMVWD3JFKQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.509982525 (84h41m6.232545565s) [noise=true , address=104.36.82.52:4087 , piece_id=YYOOIGB3SMO4QVW6PAZGO5HQANOECOZHBF77RJHRZCDZ3VF42RBQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.510173570 (84h41m6.2327366s) [noise=true , address=104.36.82.12:4164 , piece_id=P4YLWJZROD3CD2Q7M4U6ALX5Q2FCEJRKZEUWL2OUZUZJXBZSGS6A, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.510370828 (84h41m6.232933868s) [noise=true , address=104.36.82.10:4102 , piece_id=RYUNXCVFIBNYQKWVD6S7HTIREIP3AODZ3TUFGR7LQORA5IGR42FQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.511970599 (84h41m6.234533649s) [noise=true , address=104.36.82.50:4031 , piece_id=SXU73R2KHC2LAC2A3WELML3JBNHXVSQ4ZMO5U6X4UFE64AU22SHA, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.512187784 (84h41m6.234750824s) [noise=true , address=104.36.82.11:4141 , piece_id=KIAWZXCHYMGOKI4VMNNXJY7ONY6U5MLKSKARKDMD5FSJUSBVO6IQ, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:56.512474232 (84h41m6.235037272s) [noise=true , address=104.36.82.19:4420 , piece_id=TYQBC23IBRH6VXVZ2WQW2SDRUB4KKD6T3KCUQJYJ5XKQN63IDV7A, seg_pos={0 0}, upload=81277]: Uploading piece... Jul 28 00:16:57.647297514 (84h41m7.369860544s) [upload=81277]: Next returned nil segment Jul 28 00:16:57.647316822 (84h41m7.369879862s) [upload=81277]: Waiting for error group managing segments... Jul 28 00:16:58.191648108 (84h41m7.914211148s) [noise=true , address=79.127.220.98:4404 , piece_id=IKZYNE7NQIF5CLRFX77WTOWTNTYB6JNFKXNNFA33LEVFD2T3BR4A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: <nil> Jul 28 00:16:58.265274854 (84h41m7.987837894s) [noise=true , address=64.49.13.138:4483 , piece_id=UVACB7ENNG44VDUYUBV7RFPCZRNWM25LFHYM5UAGZYW7Y62MI7GQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: <nil> Jul 28 00:17:55.420737197 (84h42m5.143300237s) [noise=true , address=104.36.82.53:4145 , piece_id=V5DESVGF3MZF3Z7PSKJTLGANBLEYMQQMNQQRUZFUFBEMVWD3JFKQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 188.166.225.170:57066->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420783546 (84h42m5.143346586s) [noise=true , address=104.36.82.50:4031 , piece_id=SXU73R2KHC2LAC2A3WELML3JBNHXVSQ4ZMO5U6X4UFE64AU22SHA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12Ypc3zkMzWs3wqx9QaExy7sHvnSatwYH8Y9rHi7qt3xRELfsAa, address:104.36.82.50:4031): protocol: write tcp 188.166.225.170:39058->104.36.82.50:4031: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420815597 (84h42m5.143378637s) [noise=true , address=104.36.82.6:4893 , piece_id=D2SECKZBPZLUL4VRRIPNAWJZXPPT4HB46RXUGJYHYK5PFB7E5JSQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1259j7aHsLDMPtRwgXpd6u8axPEuaXzhRzqYYSgbpDnDqbhfwzi, address:104.36.82.6:4893): protocol: write tcp 188.166.225.170:50916->104.36.82.6:4893: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420848761 (84h42m5.143411801s) [noise=true , address=104.36.82.40:4968 , piece_id=QMTLEGLL6LWCBZP7NWKTHBZJM77O6UIA4STGEVD6OG2YNMZWM3OQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1J98jLqn1SVMYjTYxKnNqYbRxSQ5vaUnq2G7XpVngHTaWRJ8oX, address:104.36.82.40:4968): protocol: write tcp 188.166.225.170:53122->104.36.82.40:4968: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420882876 (84h42m5.143445916s) [noise=true , address=104.36.82.11:4141 , piece_id=KIAWZXCHYMGOKI4VMNNXJY7ONY6U5MLKSKARKDMD5FSJUSBVO6IQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:56046->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420916250 (84h42m5.14347929s) [noise=true , address=104.36.82.10:4102 , piece_id=RYUNXCVFIBNYQKWVD6S7HTIREIP3AODZ3TUFGR7LQORA5IGR42FQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1JEXRhcy1TphHqJXcd6qDv5gCdr6W2nJUcGoarZbsGt3TrtePy, address:104.36.82.10:4102): protocol: write tcp 188.166.225.170:34238->104.36.82.10:4102: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420948652 (84h42m5.143511692s) [noise=true , address=104.36.82.51:4071 , piece_id=NFCGPCAKZVP3UXSSKQNJLJLAI5DVLPQRGLQVN4FZ7TDJZ4EZPXQQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1VqztUNDN3hhJAMvCJtPreW7BdtAWSofQxAtmDKfDiX5sNEfWY, address:104.36.82.51:4071): protocol: rpc: dial tcp 104.36.82.51:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420987597 (84h42m5.143550627s) [noise=true , address=104.36.82.14:4237 , piece_id=BUL7F2XHF3VJUQAAAOBTZU5KUIMFHYVOKJHI3IEPXKAJWD5XORRQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1o2jLTwKNSjxjFEmckxkG1c5EuySK3o3sawGEikd2Ttz4CWfLa, address:104.36.82.14:4237): protocol: write tcp 188.166.225.170:54398->104.36.82.14:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.420997375 (84h42m5.143560415s) [noise=true , address=104.36.82.5:4847 , piece_id=XZQJ6QRZZWZ7K6ZEY4YUJUZTJNQGSKKX3GXBEE3YMZDJ7J4ARZZQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12aAq1Axcts5JgFR2ve4AmPACyCprGsyzM1PERjTYeNM4xeBH1i, address:104.36.82.5:4847): protocol: write tcp 188.166.225.170:35634->104.36.82.5:4847: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421036820 (84h42m5.14359986s) [noise=true , address=104.36.82.3:4799 , piece_id=MRKYWXL65FOM65XWLPDUFAZE7RSMNGE2I3AFFF6KH7GDDYB75FTA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:37732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421052110 (84h42m5.14361515s) [noise=true , address=104.36.82.55:4213 , piece_id=WHJCETV7HXWYPI5DJRSG7WKVMTINQ5M4G5BRFD4SQMCDNTZU6DMA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:129LRnDm7b4Fo5db7pto8GXYEtxHz9qdPAtQvaYH6gqdkMatQ6g, address:104.36.82.55:4213): protocol: write tcp 188.166.225.170:36396->104.36.82.55:4213: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421080444 (84h42m5.143643484s) [noise=true , address=104.36.82.38:4720 , piece_id=QBD4TPG3ERF66Y6R4VTT2BICWRIOFGQU3BBGQIQV3COU7YXD2J5Q, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC, address:104.36.82.38:4720): protocol: write tcp 188.166.225.170:58888->104.36.82.38:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421106795 (84h42m5.143669835s) [noise=true , address=104.36.82.52:4087 , piece_id=YYOOIGB3SMO4QVW6PAZGO5HQANOECOZHBF77RJHRZCDZ3VF42RBQ, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR, address:104.36.82.52:4087): protocol: write tcp 188.166.225.170:52952->104.36.82.52:4087: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421149516 (84h42m5.143712556s) [noise=true , address=104.36.82.56:4258 , piece_id=A5STJ7KZVBDOUMVSEYXRA7FFKTWEQDWZMZDWGEZZMV6G5BD6D7MA, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12g7DdMjQHyYTD9hRvixfNedxyPr6q6uE2rLdEbhQ1MGANRMDm3, address:104.36.82.56:4258): protocol: write tcp 188.166.225.170:35374->104.36.82.56:4258: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421155238 (84h42m5.143718288s) [noise=true , address=104.36.82.12:4164 , piece_id=P4YLWJZROD3CD2Q7M4U6ALX5Q2FCEJRKZEUWL2OUZUZJXBZSGS6A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: write tcp 188.166.225.170:33080->104.36.82.12:4164: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421160457 (84h42m5.143723497s) [noise=true , address=104.36.82.19:4420 , piece_id=TYQBC23IBRH6VXVZ2WQW2SDRUB4KKD6T3KCUQJYJ5XKQN63IDV7A, seg_pos={0 0}, upload=81277]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:104.36.82.19:4420): protocol: write tcp 188.166.225.170:42814->104.36.82.19:4420: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:17:55.421176678 (84h42m5.143739718s) [seg_pos={0 0}, upload=81277]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:17:55.652349517 (84h42m5.374912557s) [upload=81277]: Done uploading segments... Jul 28 00:17:55.652384715 (84h42m5.374947755s) [upload=81277]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234917607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 5.841081653s FAIL (2025-07-28 00:17:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841103946s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X............X........] duration: 1.919648463s FAIL (2025-07-28 00:17:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.919680293s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [................................................X...XXXXXXXX] duration: 1m0.237180814s FAIL (2025-07-28 00:16:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== ul 28 00:15:56.038704237 (84h40m5.761267277s) [noise=true , address=104.36.82.53:4154 , piece_id=P7TTDUC6PM2KWCQCRCCILAKQ4HOX5F32BBD6SGY5ZP6GT4UJ3U3Q, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.038923066 (84h40m5.761486096s) [noise=true , address=104.36.82.56:4256 , piece_id=DV6NLET3AVZGL462JHC6RMRDM63GVDNSK7ZEJQD3N4RBV2QEDD4A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039139620 (84h40m5.76170266s) [noise=true , address=104.36.82.55:4205 , piece_id=66YVGGZEASCZS6IHWXCDC6DH67LR3NDQ2IJCBL67W3NKH7LYKP4A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039152615 (84h40m5.761715655s) [noise=true , address=104.36.82.22:4518 , piece_id=CA7NKROF666BTCMCNVW2K6HXWSQRBVIJP73H6FBBCURLELMVAADQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039366424 (84h40m5.761929464s) [noise=true , address=104.36.82.11:4120 , piece_id=LNPKRA6D2O75O44CBPJKKER6ZE2Z52VXYXPV66KAUMOOMQUH5M6Q, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.039478889 (84h40m5.762041929s) [noise=true , address=104.36.82.9:4021 , piece_id=WRF3OMNR77GKWH5HNDGLQVQ4NS3CHSO623KUK3GF3BAXQCGKBJEQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042284787 (84h40m5.764847827s) [noise=true , address=104.36.82.47:4532 , piece_id=KBYTMZOSQVJOUUM47VNFO5TRW6UJUVQQPUMWXP5P424A6PWQJZHQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042642261 (84h40m5.765205301s) [noise=true , address=104.36.82.10:4101 , piece_id=LT53BM6ZPX7HXCKXTJSO7L4TSMRLOZNCVS3ODWJBY75553AL373A, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.042922227 (84h40m5.765485267s) [noise=true , address=64.49.13.140:4509 , piece_id=IYFIKRLJS6LLQ6T4ECNFTNI5FH4RP3GAK3AZOOIEIC6L4W7DSRTQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:56.043238372 (84h40m5.765801412s) [noise=true , address=104.36.82.54:4163 , piece_id=HH4JNYZES6QTFEWTMG26X3KSTKGUCZLZXNGVF6B3MARAXEN333HQ, seg_pos={0 0}, upload=81264]: Uploading piece... Jul 28 00:15:57.198595100 (84h40m6.92115814s) [upload=81264]: Next returned nil segment Jul 28 00:15:57.198610891 (84h40m6.921173931s) [upload=81264]: Waiting for error group managing segments... Jul 28 00:15:57.690628138 (84h40m7.413191178s) [noise=true , address=64.49.13.140:4509 , piece_id=IYFIKRLJS6LLQ6T4ECNFTNI5FH4RP3GAK3AZOOIEIC6L4W7DSRTQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:15:58.353679667 (84h40m8.076242708s) [noise=true , address=64.49.13.138:4482 , piece_id=IAOXAL7D4JGTLA3YME7UZADCHYFHFLZYPBO4WV7NN7F55P366MNA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:15:59.537815957 (84h40m9.260378997s) [noise=true , address=79.127.201.215:4346 , piece_id=C5JV4YHK5T2EJPNFP544PSVFJ34PKGRYE35BCE3TSN7EUYPAE5AQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: <nil> Jul 28 00:16:54.937395756 (84h41m4.659958797s) [noise=true , address=104.36.82.15:4277 , piece_id=DUMG7WOWJPYRH3OJIMGEJYI3FVCLU2ZA2LWEVNLRDJFBBREBFUSQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937446704 (84h41m4.660009734s) [noise=true , address=104.36.82.55:4197 , piece_id=DGDTNLKYOEWE5RI27BVDY5PQZC64ANIY2DQL7VZSG4KJDWOR53DA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: rpc: dial tcp 104.36.82.55:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937477763 (84h41m4.660040813s) [noise=true , address=104.36.82.46:4490 , piece_id=P7YYMWSRWBN2C6PE52FLZY5GGWZXCLBVBAQS77JTUV6E63UMLXFQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji, address:104.36.82.46:4490): protocol: rpc: dial tcp 104.36.82.46:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937510205 (84h41m4.660073245s) [noise=true , address=104.36.82.53:4154 , piece_id=P7TTDUC6PM2KWCQCRCCILAKQ4HOX5F32BBD6SGY5ZP6GT4UJ3U3Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937576563 (84h41m4.660139603s) [noise=true , address=104.36.82.55:4205 , piece_id=66YVGGZEASCZS6IHWXCDC6DH67LR3NDQ2IJCBL67W3NKH7LYKP4A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:104.36.82.55:4205): protocol: rpc: dial tcp 104.36.82.55:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937642859 (84h41m4.660205899s) [noise=true , address=104.36.82.4:4813 , piece_id=OJUKKE66EHBQOEOQQ4CKWNOFAXMESJP7YVUNK73FZRK7OG6WOKIA, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12EHHTcQmhaHswSx1ubyHmPTHBB5ptuxmydbKHLjYChF1mudMSB, address:104.36.82.4:4813): protocol: write tcp 188.166.225.170:48730->104.36.82.4:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937679138 (84h41m4.660242178s) [noise=true , address=104.36.82.50:4015 , piece_id=DBKNZBUPKKWIITIGDWU4CHCLQ2IS3PAMUVBKWLD47GSBIIEXUV4Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:104.36.82.50:4015): protocol: rpc: dial tcp 104.36.82.50:4015: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937716340 (84h41m4.66027937s) [noise=true , address=104.36.82.36:4634 , piece_id=I24D6SC22IPDGYQHC5MR7N4H4UXPNSUGFC2NMXOWCWID3U4GXJOQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1kwydp3q2wTrCVadJZB7WmKGK8h2R5QL7HGpUsjTpRbcnWUSb5, address:104.36.82.36:4634): protocol: rpc: dial tcp 104.36.82.36:4634: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937746518 (84h41m4.660309558s) [noise=true , address=104.36.82.10:4101 , piece_id=LT53BM6ZPX7HXCKXTJSO7L4TSMRLOZNCVS3ODWJBY75553AL373A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: rpc: dial tcp 104.36.82.10:4101: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937838624 (84h41m4.660401664s) [noise=true , address=104.36.82.54:4163 , piece_id=HH4JNYZES6QTFEWTMG26X3KSTKGUCZLZXNGVF6B3MARAXEN333HQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12CgPQ5hkrJdqbJvGarxKjXvYYnrXQqapL5BH2rNJNvnhpRi3pw, address:104.36.82.54:4163): protocol: write tcp 188.166.225.170:56394->104.36.82.54:4163: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937879693 (84h41m4.660442733s) [noise=true , address=104.36.82.56:4256 , piece_id=DV6NLET3AVZGL462JHC6RMRDM63GVDNSK7ZEJQD3N4RBV2QEDD4A, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937901905 (84h41m4.660464935s) [noise=true , address=104.36.82.9:4021 , piece_id=WRF3OMNR77GKWH5HNDGLQVQ4NS3CHSO623KUK3GF3BAXQCGKBJEQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg, address:104.36.82.9:4021): protocol: rpc: dial tcp 104.36.82.9:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937918316 (84h41m4.660481356s) [noise=true , address=104.36.82.22:4518 , piece_id=CA7NKROF666BTCMCNVW2K6HXWSQRBVIJP73H6FBBCURLELMVAADQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:1ozLDHDa9ankBimUBjpM91VrraKMptSCQPoDS9yWu2tjVErasU, address:104.36.82.22:4518): protocol: rpc: dial tcp 104.36.82.22:4518: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937946160 (84h41m4.6605092s) [noise=true , address=104.36.82.47:4532 , piece_id=KBYTMZOSQVJOUUM47VNFO5TRW6UJUVQQPUMWXP5P424A6PWQJZHQ, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937972520 (84h41m4.66053556s) [noise=true , address=104.36.82.11:4120 , piece_id=LNPKRA6D2O75O44CBPJKKER6ZE2Z52VXYXPV66KAUMOOMQUH5M6Q, seg_pos={0 0}, upload=81264]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: write tcp 188.166.225.170:55330->104.36.82.11:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:16:54.937998750 (84h41m4.66056179s) [seg_pos={0 0}, upload=81264]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:16:55.172427108 (84h41m4.894990148s) [upload=81264]: Done uploading segments... Jul 28 00:16:55.172457186 (84h41m4.895020226s) [upload=81264]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.237213348s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 5.831422759s FAIL (2025-07-28 00:16:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.83144999s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.008775427s FAIL (2025-07-28 00:16:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.00880365s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X............X.......] duration: 1.905069773s FAIL (2025-07-28 00:16:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.905096363s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 5.876948712s FAIL (2025-07-28 00:15:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.876972637s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [.................................................X...XXXXXXX] duration: 1m0.229436487s FAIL (2025-07-28 00:15:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== : Uploading piece... Jul 28 00:14:53.331896307 (84h39m3.054459347s) [noise=true , address=104.36.82.46:4488 , piece_id=4S57UPCVJQGDDJW6EEJMKTEJNDXUUDCX4UCDWTROVDUPF6TVOLUA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332130846 (84h39m3.054693886s) [noise=true , address=104.36.82.42:4812 , piece_id=KHVC7T4LZBS5EFCM3NUY3BGAGLYKSSXHIFADD3XZ57SOLXM4SUJA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332396654 (84h39m3.054959694s) [noise=true , address=104.36.82.9:4032 , piece_id=GNBIEEQ2FDSZYIHGASPQ5QHYMMK7PQTM7URHIHBWQDLECI6SQRSA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332618228 (84h39m3.055181268s) [noise=true , address=104.36.82.4:4828 , piece_id=EZIFSYBIGFQFTCNMFFGHG5XYIT2UO6SVMZBH4PUGMPZDHCE3VVVQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.332842558 (84h39m3.055405588s) [noise=true , address=104.36.82.49:4990 , piece_id=FCXU7726KRDGV2ZMIWOCMNFO2PRBY4GAFLXK6PLV6KM2XODFYYBQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333070313 (84h39m3.055633353s) [noise=true , address=104.36.82.15:4279 , piece_id=R4IDM5KBBZKVLKDVFCTUH4FYI7JHJEEVWEXMXCZE3OEV5DCT5CJQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333287098 (84h39m3.055850128s) [noise=true , address=104.36.82.13:4188 , piece_id=K2ZWCGD2ZJU7RL6Z3NV3LZ3Z7OA62F6ULRUWDHGVWEGLUOZU4IXA, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:53.333504775 (84h39m3.056067815s) [noise=true , address=104.36.82.55:4201 , piece_id=MRRUQF7KYU3ZDKWEYDSMTAYY5HFSFY6UFFW7TSAP2J6GDPITG2AQ, seg_pos={0 0}, upload=81249]: Uploading piece... Jul 28 00:14:54.841644854 (84h39m4.564207894s) [upload=81249]: Next returned nil segment Jul 28 00:14:54.841663019 (84h39m4.564226059s) [upload=81249]: Waiting for error group managing segments... Jul 28 00:14:55.226766917 (84h39m4.949329957s) [noise=true , address=79.127.201.214:4291 , piece_id=MAF5NZGDQY64EMZXE7KTO76PQLE4EDAGY642BXCKC4FWFKFGCPEA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: <nil> Jul 28 00:14:56.329867134 (84h39m6.052430175s) [noise=true , address=64.49.13.140:4508 , piece_id=6INZQBFHYBJFICMBZB6ZNNH225TO6LRZ65JTAJRBLD6KE3ZJCOVA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: <nil> Jul 28 00:15:52.240213723 (84h40m1.962776763s) [noise=true , address=104.36.82.5:4842 , piece_id=BQACPUFGZCKZHCTZRRXE77KNMIFLOVCJIGHDJGV3WDYPWNYQB6SQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240287545 (84h40m1.962850585s) [noise=true , address=104.36.82.47:4528 , piece_id=DULYM42F56QCMRCAJPQHC4I3SNMQE3YEDA5UEEY7TCEJ2SXSBWVQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: rpc: dial tcp 104.36.82.47:4528: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240507244 (84h40m1.963070284s) [noise=true , address=104.36.82.14:4240 , piece_id=RT47AXFOZSGNOOAGUHG6U4BBRNL3ZTTWMAS7OEZ33F7RQPVDTAHA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240566037 (84h40m1.963129108s) [noise=true , address=104.36.82.12:4184 , piece_id=O2OHDSYBKHHF3ZCSVBYEHGVDO3N24BFXWUNPUDELV52ZACXOCFHA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:17JJjwwgSn7A6BS7n1a7fphX8oTEQpXmad25ifM2nDhWhPv4Ls, address:104.36.82.12:4184): protocol: rpc: dial tcp 104.36.82.12:4184: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240616984 (84h40m1.963180034s) [noise=true , address=104.36.82.4:4828 , piece_id=EZIFSYBIGFQFTCNMFFGHG5XYIT2UO6SVMZBH4PUGMPZDHCE3VVVQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:46712->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240685956 (84h40m1.963248997s) [noise=true , address=104.36.82.42:4812 , piece_id=KHVC7T4LZBS5EFCM3NUY3BGAGLYKSSXHIFADD3XZ57SOLXM4SUJA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1TmXeVTeejZvrWi2RPxMutM6kXuZiMCtPxQBwL4tq6mhoLwFQo, address:104.36.82.42:4812): protocol: write tcp 188.166.225.170:53524->104.36.82.42:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240730933 (84h40m1.963293973s) [noise=true , address=104.36.82.46:4502 , piece_id=IK6H4XUBDSC27HNUSCAYS7D3OXBMCLIARPPCFTMB6H5XDS225DCA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.240886610 (84h40m1.96344965s) [noise=true , address=104.36.82.3:4799 , piece_id=VMYQUXIGXCPHFFPAYMBTIBKQ6SDN62NO6IU2C66RESUUHOZNNRTQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: rpc: dial tcp 104.36.82.3:4799: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241047739 (84h40m1.963610779s) [noise=true , address=104.36.82.45:4936 , piece_id=PCTADNWF4UUA27YJBIRMNWCGHOIRKA7ZGRNAH4DPQHSMI72NOBGA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1DicJh1hd5NUoj9uxoqzeaCmjGHwfk2VUaA9zQFmv72wBdBoW8, address:104.36.82.45:4936): protocol: rpc: dial tcp 104.36.82.45:4936: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241141237 (84h40m1.963704287s) [noise=true , address=104.36.82.15:4279 , piece_id=R4IDM5KBBZKVLKDVFCTUH4FYI7JHJEEVWEXMXCZE3OEV5DCT5CJQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12uiA6tmZtmHDNaNtkhRtH31q4MJxn3C4bfrLAXmYz5mKmofHoG, address:104.36.82.15:4279): protocol: write tcp 188.166.225.170:56188->104.36.82.15:4279: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241230919 (84h40m1.963793959s) [noise=true , address=104.36.82.9:4032 , piece_id=GNBIEEQ2FDSZYIHGASPQ5QHYMMK7PQTM7URHIHBWQDLECI6SQRSA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1ferXUYkCNRALvKCfTAfqHrNuDJZcqAx7Qx3hr7TaaMdVGzxQQ, address:104.36.82.9:4032): protocol: write tcp 188.166.225.170:52692->104.36.82.9:4032: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241269803 (84h40m1.963832853s) [noise=true , address=104.36.82.54:4182 , piece_id=7JBUEBASEJEFJ7HIIU2OED56QSN4REZEUZLAMXQNTEVHOXEMPMXQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12HbkH31sfDUnZLXAbmzQphcFYmzB8MQo98Q6Ufbo59et42zxMy, address:104.36.82.54:4182): protocol: rpc: dial tcp 104.36.82.54:4182: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241310532 (84h40m1.963873582s) [noise=true , address=104.36.82.13:4188 , piece_id=K2ZWCGD2ZJU7RL6Z3NV3LZ3Z7OA62F6ULRUWDHGVWEGLUOZU4IXA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12pLFHQ2Xj6n53w9eLQtMB4A29k3Xh6w716z5s2CGxwPnBoZcgZ, address:104.36.82.13:4188): protocol: rpc: dial tcp 104.36.82.13:4188: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241319639 (84h40m1.963882679s) [noise=true , address=104.36.82.46:4488 , piece_id=4S57UPCVJQGDDJW6EEJMKTEJNDXUUDCX4UCDWTROVDUPF6TVOLUA, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 188.166.225.170:35244->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241353343 (84h40m1.963916393s) [noise=true , address=104.36.82.49:4990 , piece_id=FCXU7726KRDGV2ZMIWOCMNFO2PRBY4GAFLXK6PLV6KM2XODFYYBQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:1gDzCKcMaLTisHw68GYkMbcMJF7rbKnUt48etS8Q7KeiXedbU4, address:104.36.82.49:4990): protocol: write tcp 188.166.225.170:35316->104.36.82.49:4990: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241372310 (84h40m1.96393535s) [noise=true , address=104.36.82.55:4201 , piece_id=MRRUQF7KYU3ZDKWEYDSMTAYY5HFSFY6UFFW7TSAP2J6GDPITG2AQ, seg_pos={0 0}, upload=81249]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: write tcp 188.166.225.170:56302->104.36.82.55:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:52.241392449 (84h40m1.963955499s) [seg_pos={0 0}, upload=81249]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:15:52.467954056 (84h40m2.190517106s) [upload=81249]: Done uploading segments... Jul 28 00:15:52.467984765 (84h40m2.190547805s) [upload=81249]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.229450765s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.234973806s FAIL (2025-07-28 00:15:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 4h38m49.468321249s) [noise=true , address=45.91.116.12:4225 , piece_id=7O4KTDOCFH2LH42FZN6UB56MO42UADU4VPZL3WV76ATOAIN3T4XQ, seg_pos={0 0}, upload=81244]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Jul 28 00:14:39.745767448 (84h38m49.468330488s) [seg_pos={0 0}, upload=81244]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:14:39.745916893 (84h38m49.468479933s) [upload=81244]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:14:39.745921191 (84h38m49.468484231s) [upload=81244]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:14:40.037599193 (84h38m49.760162233s) [upload=81244]: Done uploading segments... Jul 28 00:14:40.037615805 (84h38m49.760178845s) [upload=81244]: Upload finished. err: <nil> Jul 28 00:15:04.480798899 (84h39m14.20336194s) [upload=81252]: Starting upload Jul 28 00:15:04.480876769 (84h39m14.203439839s) [upload=81252]: Done starting upload Jul 28 00:15:04.481001578 (84h39m14.203564628s) [upload=81252]: Uploading segments... Jul 28 00:15:04.481081461 (84h39m14.203644501s) [upload=81252]: Got next segment. Inline: false Jul 28 00:15:04.481113221 (84h39m14.203676282s) [upload=81252]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:15:04.481119724 (84h39m14.203682764s) [upload=81252]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:15:04.716839411 (84h39m14.439402461s) [seg_pos={0 0}, upload=81252]: Begin upload segment... Jul 28 00:15:04.716900478 (84h39m14.439463528s) [seg_pos={0 0}, upload=81252]: Done begin upload segment. Jul 28 00:15:04.716928301 (84h39m14.439491351s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.716959711 (84h39m14.439522751s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717364165 (84h39m14.439927205s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717377661 (84h39m14.439940701s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717621617 (84h39m14.440184658s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717695910 (84h39m14.44025895s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.717898938 (84h39m14.440461978s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718043215 (84h39m14.440606255s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718351064 (84h39m14.440914114s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718650747 (84h39m14.441213787s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.718948056 (84h39m14.441511096s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719239924 (84h39m14.441802974s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719543695 (84h39m14.442106735s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.719846184 (84h39m14.442409224s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720157299 (84h39m14.442720339s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:04.720484405 (84h39m14.443047455s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Uploading piece... Jul 28 00:15:06.449685788 (84h39m16.172248828s) [upload=81252]: Next returned nil segment Jul 28 00:15:06.449705004 (84h39m16.172268084s) [upload=81252]: Waiting for error group managing segments... Jul 28 00:15:06.740940750 (84h39m16.46350378s) [noise=true , address=77.90.31.18:4807 , piece_id=SB2EOKTVXXPHDQQEHB4YJBBXLOXEBQN7KQSRBOVZP42ZGIUSIU7Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.741446858 (84h39m16.464009898s) [noise=true , address=77.90.31.12:4592 , piece_id=LH4YNTF4IXLG3CVWSUIDXX2XTOEFMPCFEFD3RYST6IFO7VCGNBKQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.747334472 (84h39m16.469897542s) [noise=true , address=45.91.116.6:4973 , piece_id=Q4QGSWOSVLFAGBJVJ6MY3WBQSRVPUVGAKOBBWQXD2KTJWX7JXXNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.748805717 (84h39m16.471368747s) [noise=true , address=45.91.116.10:4081 , piece_id=6ODMN4RNRD577D7Y2IW7VAQYWXSLUX4A2VDJRDRUTMSHAUNUSNZQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.815890697 (84h39m16.538453737s) [noise=true , address=45.91.116.4:4937 , piece_id=RTPEK3HBRIWLZNQRXNKFGD34GGBR5QX4FHYQUFOUJHN23XX2OZCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.819563003 (84h39m16.542126043s) [noise=true , address=77.90.31.21:4877 , piece_id=6FMP75OPY5ROWCJKEDRKVTCONJ3GTHEQTYLQG2SPWEK7BWNIP5VQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.830587626 (84h39m16.553150666s) [noise=true , address=45.91.116.5:4299 , piece_id=DKCLRZ5J3GKZHBMK3IALNT35KH7XCMCLONPJDYP3YFAMXFK5E3CQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.842458518 (84h39m16.565021558s) [noise=true , address=77.90.31.13:4627 , piece_id=7OTR55UTQONXNMRT5LM43BN7W5HYH5KHCLSSMOHU33AF2YJ4JUCA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.846136164 (84h39m16.568699204s) [noise=true , address=45.91.116.6:4975 , piece_id=K2GNEYG5JIEDQXQNB7RYVK7UAWOZ3RTO3U3AZPGH6FLRUE3HISOQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.857847821 (84h39m16.580410861s) [noise=true , address=45.91.116.14:4189 , piece_id=7FBJZBG5CJDWWS4TGKJATJSSBLSPZXPRNOOPHIPYMLKPKTHYMTQQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.892512974 (84h39m16.615076035s) [noise=true , address=45.91.116.7:4010 , piece_id=IDN2SM6QUYUYNTU4QZXULENX63SP3DE7YLOZ5V2XGFS7TPLZDD3A, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:06.898154658 (84h39m16.620717729s) [noise=true , address=45.91.116.12:4226 , piece_id=XQYIJYMKLGOTI5NHR7TE7FW5P54KBQSIM5RVIS3JZUO5EBNGU7QQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: <nil> Jul 28 00:15:33.823200918 (84h39m43.545763958s) [noise=true , address=104.36.82.16:4293 , piece_id=ZWJUPUSTFIPB6FGUUWOOVIYXNCWPUF567DF6327BU76FW2EMYKNA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823269078 (84h39m43.545832118s) [noise=true , address=104.36.82.8:4970 , piece_id=OQBN2SMVR7DCWABX75PHXBVT2JFH4E336LQHXQDHYR5HXDNA6E4Q, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823301911 (84h39m43.545864951s) [noise=true , address=104.36.82.7:4911 , piece_id=JNMBXPYUEDWN3WNCQCQHR4B7DCIDDRX22GW57NEP543OWYYEWIZA, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823310658 (84h39m43.545873728s) [noise=true , address=104.36.82.11:4113 , piece_id=7BSZUMUDAUNNYNVMVXFTOYD7CTWVSRCLCZGOMP3QJTE547M5HXVQ, seg_pos={0 0}, upload=81252]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 188.166.225.170:41026->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:15:33.823344703 (84h39m43.545907743s) [seg_pos={0 0}, upload=81252]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:15:34.056565111 (84h39m43.779128151s) [upload=81252]: Done uploading segments... Jul 28 00:15:34.056591671 (84h39m43.779154711s) [upload=81252]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.23499723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X.......................................X............X......] duration: 1.916787595s FAIL (2025-07-28 00:15:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.916814297s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.878546525s FAIL (2025-07-28 00:15:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.878568186s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [..................................................X...XXXXXX] duration: 1m0.232662726s FAIL (2025-07-28 00:14:51 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== dress=104.36.82.15:4272 , piece_id=ATORNVQOPR2HQKXDUK4HTSM6T63XI4HPQ4MKPGDNCQNF2OYB72OQ, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.862313504 (84h38m2.584876545s) [noise=true , address=104.36.82.45:4931 , piece_id=LGPKF5KPJTFQA3XEWAHKINOJVIUPO3EPFNPFMRQTJBPE3C5BJ5NQ, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.862589813 (84h38m2.585152853s) [noise=true , address=104.36.82.20:4459 , piece_id=BHOSANJUYPWEJ3ZFURC3SYLWLFDTCPJADGUFV2WHAMZ7VKODXUVQ, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.862877784 (84h38m2.585440814s) [noise=true , address=104.36.82.3:4780 , piece_id=7FHRJKC6CA7L7TYFP2Y3JGQIGLXLQNYCAZ2ISAHOOXOKVNHGZKPA, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.863136730 (84h38m2.58569977s) [noise=true , address=104.36.82.34:4570 , piece_id=K65XDI4VBHLTVZIHIDNLBKLDHI7PHOGJQKTWV6NCVVBV5XC4BOGA, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.863389633 (84h38m2.585952673s) [noise=true , address=104.36.82.52:4106 , piece_id=YVYHK7B2REVZKQRL36M5TRVUVVFKTK72YJGZXQ6Q3FLSE5BRTIVQ, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.863655061 (84h38m2.586218102s) [noise=true , address=104.36.82.12:4155 , piece_id=4QEV4D7WNE7GMRIXL54LNE3VMBDEKNGELIKOJ7V7335UGS6P622Q, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.863889399 (84h38m2.586452439s) [noise=true , address=104.36.82.42:4826 , piece_id=7SLWMPYIFWXV3MM4BCG4L7LCT5GEQZOUNYWUZ3M2NPX37QRQHWHA, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:13:52.864083921 (84h38m2.586646951s) [noise=true , address=104.36.82.16:4316 , piece_id=JCFIK5GKUG4D2L3BOFE43SG7WVDUJ5O2S6IRZQWQBS7TIMIFXYOQ, seg_pos={0 0}, upload=81230]: Uploading piece... Jul 28 00:14:51.766579194 (84h39m1.489142234s) [upload=81230]: Next segment err: context deadline exceeded Jul 28 00:14:51.768167593 (84h39m1.490730633s) [noise=true , address=104.36.82.42:4826 , piece_id=7SLWMPYIFWXV3MM4BCG4L7LCT5GEQZOUNYWUZ3M2NPX37QRQHWHA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:13KQwy5RDTnWanKbCdnsdqRt88m58bEqcLmT3SpLxYW2uANXGv, address:104.36.82.42:4826): protocol: rpc: dial tcp 104.36.82.42:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768207530 (84h39m1.49077056s) [noise=true , address=104.36.82.49:4978 , piece_id=GZF33LD6THFYRSDEXMQRGA4I4CWVM2MHBUVLW37NFVEQ5YZTGZRA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:104.36.82.49:4978): protocol: rpc: dial tcp 104.36.82.49:4978: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768246414 (84h39m1.490809454s) [noise=true , address=104.36.82.56:4247 , piece_id=LNUPJNA3AGPNN33CGMTVNEFWGQPDZETYFGRDERB76GE3JBIEK5BA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768302802 (84h39m1.490865842s) [noise=true , address=104.36.82.16:4316 , piece_id=JCFIK5GKUG4D2L3BOFE43SG7WVDUJ5O2S6IRZQWQBS7TIMIFXYOQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12PPnXDzwXDVDKemw483GXuzQrPN48JJrjeeD7az2NPNawHNaXf, address:104.36.82.16:4316): protocol: rpc: dial tcp 104.36.82.16:4316: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768414616 (84h39m1.490977656s) [noise=true , address=104.36.82.3:4780 , piece_id=7FHRJKC6CA7L7TYFP2Y3JGQIGLXLQNYCAZ2ISAHOOXOKVNHGZKPA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1i8cJRxzLNvUYqELrufixk24oDZwDHroyG5Cfy13csTPgeajRM, address:104.36.82.3:4780): protocol: rpc: dial tcp 104.36.82.3:4780: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768459733 (84h39m1.491022773s) [noise=true , address=104.36.82.40:4941 , piece_id=3NYJS4FLTD64SN7PRYPFZ2O7I2AXG7NS42XAGIMBJOLQUFVHVS3Q, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni, address:104.36.82.40:4941): protocol: rpc: dial tcp 104.36.82.40:4941: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768491102 (84h39m1.491054143s) [noise=true , address=104.36.82.54:4183 , piece_id=3IGP5HITYYUZTT54PLKVJGB776M55IJYHIKAU5VRWUKU5SVCEBFA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: rpc: dial tcp 104.36.82.54:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768518735 (84h39m1.491081765s) [noise=true , address=104.36.82.15:4272 , piece_id=ATORNVQOPR2HQKXDUK4HTSM6T63XI4HPQ4MKPGDNCQNF2OYB72OQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1UeKsX3pQfhvCukBvJf1sekK7N6sXkaWaJnZzETKoxyXWo2BFi, address:104.36.82.15:4272): protocol: rpc: dial tcp 104.36.82.15:4272: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768549444 (84h39m1.491112484s) [noise=true , address=104.36.82.20:4459 , piece_id=BHOSANJUYPWEJ3ZFURC3SYLWLFDTCPJADGUFV2WHAMZ7VKODXUVQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768576315 (84h39m1.491139355s) [noise=true , address=104.36.82.52:4106 , piece_id=YVYHK7B2REVZKQRL36M5TRVUVVFKTK72YJGZXQ6Q3FLSE5BRTIVQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768617415 (84h39m1.491180455s) [noise=true , address=104.36.82.12:4155 , piece_id=4QEV4D7WNE7GMRIXL54LNE3VMBDEKNGELIKOJ7V7335UGS6P622Q, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:104.36.82.12:4155): protocol: rpc: dial tcp 104.36.82.12:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768646751 (84h39m1.491209791s) [noise=true , address=104.36.82.34:4570 , piece_id=K65XDI4VBHLTVZIHIDNLBKLDHI7PHOGJQKTWV6NCVVBV5XC4BOGA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768673532 (84h39m1.491236572s) [noise=true , address=104.36.82.4:4807 , piece_id=Q77HUDQ5USUNQBRZ7M4EEJIOHDQE6ZND2MRYAZD5TPCOH75MX5PA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1MZ9LJLHF4XXvmRG2r7XkanDfdNSKCUgua9EBPusPuGyLzTLo8, address:104.36.82.4:4807): protocol: rpc: dial tcp 104.36.82.4:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768707727 (84h39m1.491270767s) [noise=true , address=104.36.82.10:4094 , piece_id=XX5DTHANKBQCZRUTG3ODR5F66EAYI5EVKEKXR24N2GL65BAONXHQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: rpc: dial tcp 104.36.82.10:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768749697 (84h39m1.491312737s) [noise=true , address=104.36.82.55:4200 , piece_id=PEGRYWJI3I7YMXX3IAS2S3MR6A4ATWTQAZ2G4QSOZPGI46DU24PA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768781047 (84h39m1.491344087s) [noise=true , address=104.36.82.5:4842 , piece_id=OGG3HNTIGP3PZZCQJXTZTBTFS3D5ATRB3HMGOCWJQCWZ3U3C4LWA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768818800 (84h39m1.49138184s) [noise=true , address=104.36.82.46:4514 , piece_id=LXAOL43JK64VTL72UO5IBKJHMCGRTGUABOSPXHPXYAK7KPANRMNA, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: rpc: dial tcp 104.36.82.46:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768851492 (84h39m1.491414532s) [noise=true , address=104.36.82.45:4931 , piece_id=LGPKF5KPJTFQA3XEWAHKINOJVIUPO3EPFNPFMRQTJBPE3C5BJ5NQ, seg_pos={0 0}, upload=81230]: Done uploading piece. err: ecclient: upload failed (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu, address:104.36.82.45:4931): protocol: write tcp 188.166.225.170:43064->104.36.82.45:4931: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:51.768861521 (84h39m1.491424561s) [seg_pos={0 0}, upload=81230]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.232681782s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.814374854s FAIL (2025-07-28 00:14:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== onn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:02.541185363 (84h37m12.263748413s) [seg_pos={0 0}, upload=81217]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:13:02.541379915 (84h37m12.263942965s) [upload=81217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:13:02.541387379 (84h37m12.263950429s) [upload=81217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:13:02.842765569 (84h37m12.565328619s) [upload=81217]: Done uploading segments... Jul 28 00:13:02.842818560 (84h37m12.565381611s) [upload=81217]: Upload finished. err: <nil> Jul 28 00:14:06.832083031 (84h38m16.554646071s) [upload=81234]: Starting upload Jul 28 00:14:06.832143917 (84h38m16.554706957s) [upload=81234]: Done starting upload Jul 28 00:14:06.832192500 (84h38m16.55475554s) [upload=81234]: Uploading segments... Jul 28 00:14:06.832308402 (84h38m16.554871432s) [upload=81234]: Got next segment. Inline: false Jul 28 00:14:06.832332729 (84h38m16.554895769s) [upload=81234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:14:06.832379038 (84h38m16.554942078s) [upload=81234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:14:07.069638894 (84h38m16.792201934s) [seg_pos={0 0}, upload=81234]: Begin upload segment... Jul 28 00:14:07.069780064 (84h38m16.792343114s) [seg_pos={0 0}, upload=81234]: Done begin upload segment. Jul 28 00:14:07.069815833 (84h38m16.792378863s) [noise=true , address=45.91.116.14:4191 , piece_id=EWZQOAGO2HJIOHAGPWO4I5KZYMYDH6KEWIWH7OJYZU7X6QX2PJRA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.069823998 (84h38m16.792387028s) [noise=true , address=104.36.82.16:4293 , piece_id=S44IV2VXPTE2T2S3UJZXJDXS3KF2P73RVRDK6MKQ7LXHFKY7UDQQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070154851 (84h38m16.792717891s) [noise=true , address=104.36.82.13:4186 , piece_id=SJZEBXVKECXJLBRIFQ6XJ2J3YJFXZPBNPRA426DLEALYD2U5ZIQA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070406503 (84h38m16.792969543s) [noise=true , address=45.91.116.6:4973 , piece_id=6O6O4FJA7KJ4YCGZZUJFTJTXSGEDC5NGJKP5VD354Z3ATBXGWJJA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070667613 (84h38m16.793230653s) [noise=true , address=104.36.82.4:4802 , piece_id=ETDF7XUCA2IDDPZXPQIEUJMDJSCCLDBB277HC4NB74I5DWRETDKQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071031479 (84h38m16.793594529s) [noise=true , address=45.91.116.13:4154 , piece_id=MMSBZ6S6KNGT466PK3JDFAYECE2HKAKI2C2XHUHXNMD46HEGDDTQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071287899 (84h38m16.793850939s) [noise=true , address=45.91.116.9:4263 , piece_id=BF625DXSIWXKOS6M52PYDGLXLBGL7PVQV3XJ7XPZVGZ6FPXNQVPQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071502440 (84h38m16.79406548s) [noise=true , address=45.91.116.12:4226 , piece_id=IOROHEXTPBROJPQF7HX7PHV3I4VIH5BGZ7V4MEMZAXHK6TIWJXCA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071694066 (84h38m16.794257106s) [noise=true , address=77.90.31.12:4589 , piece_id=IY2OWMCRRSCLIMNRLRHWZYZMNYHJAKMDJB55CDPZQCRGAIAILDQA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071994812 (84h38m16.794557862s) [noise=true , address=45.91.116.12:4225 , piece_id=7UQPLI645A4SJWMWEQRBKH2252N4VLJTBGGK3XQ76YCHPE4GGDTQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.072908810 (84h38m16.79547185s) [noise=true , address=77.90.31.16:4735 , piece_id=ZMU734HSUQRZBQUSXLCYWF7W4GTTB7NXUFETODPFEVFWYVZFQ6EA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.072928679 (84h38m16.795491719s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJHTDAEM23LGFK4EK4D3P6S5PMSKHGURLOQ23NR6W6TFCYTDBISA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.073356907 (84h38m16.795919948s) [noise=true , address=77.90.31.10:4518 , piece_id=UQH55LUAWYR5NSVRXXE2LBVJUZQOPRHEDUH2S6E4OY63W5KRYRUQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.073973647 (84h38m16.796536687s) [noise=true , address=104.36.82.8:4972 , piece_id=FGHWAMXGXDSFD7Y7NHC3LLEYVOLDGH725HY27RBQR5WC2JSEZ3HA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.074190893 (84h38m16.796753933s) [noise=true , address=77.90.31.19:4842 , piece_id=6O5K5OL3PZ6DHQG2OTEB3DEJJPJSKPVLX2YPAFKZNJIHRVAZHRZA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.074611729 (84h38m16.797174769s) [noise=true , address=104.36.82.5:4840 , piece_id=SWGYOU5A43UT2YEOEG2CS2TDHLGB5PCBC663AQH2D75VV4CHY2TQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:08.828150913 (84h38m18.550713954s) [upload=81234]: Next returned nil segment Jul 28 00:14:08.828193245 (84h38m18.550756285s) [upload=81234]: Waiting for error group managing segments... Jul 28 00:14:09.105320036 (84h38m18.827883076s) [noise=true , address=45.91.116.9:4263 , piece_id=BF625DXSIWXKOS6M52PYDGLXLBGL7PVQV3XJ7XPZVGZ6FPXNQVPQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.106907042 (84h38m18.829470082s) [noise=true , address=77.90.31.12:4589 , piece_id=IY2OWMCRRSCLIMNRLRHWZYZMNYHJAKMDJB55CDPZQCRGAIAILDQA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.149925576 (84h38m18.872488617s) [noise=true , address=45.91.116.13:4154 , piece_id=MMSBZ6S6KNGT466PK3JDFAYECE2HKAKI2C2XHUHXNMD46HEGDDTQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.161168177 (84h38m18.883731237s) [noise=true , address=45.91.116.14:4191 , piece_id=EWZQOAGO2HJIOHAGPWO4I5KZYMYDH6KEWIWH7OJYZU7X6QX2PJRA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.201783955 (84h38m18.924346995s) [noise=true , address=77.90.31.10:4518 , piece_id=UQH55LUAWYR5NSVRXXE2LBVJUZQOPRHEDUH2S6E4OY63W5KRYRUQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.206134097 (84h38m18.928697137s) [noise=true , address=45.91.116.12:4226 , piece_id=IOROHEXTPBROJPQF7HX7PHV3I4VIH5BGZ7V4MEMZAXHK6TIWJXCA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.211591940 (84h38m18.93415498s) [noise=true , address=77.90.31.19:4842 , piece_id=6O5K5OL3PZ6DHQG2OTEB3DEJJPJSKPVLX2YPAFKZNJIHRVAZHRZA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.214527045 (84h38m18.937090085s) [noise=true , address=45.91.116.6:4973 , piece_id=6O6O4FJA7KJ4YCGZZUJFTJTXSGEDC5NGJKP5VD354Z3ATBXGWJJA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.243601201 (84h38m18.966164241s) [noise=true , address=45.91.116.12:4225 , piece_id=7UQPLI645A4SJWMWEQRBKH2252N4VLJTBGGK3XQ76YCHPE4GGDTQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.273963521 (84h38m18.996526561s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJHTDAEM23LGFK4EK4D3P6S5PMSKHGURLOQ23NR6W6TFCYTDBISA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.362915428 (84h38m19.085478468s) [noise=true , address=77.90.31.16:4735 , piece_id=ZMU734HSUQRZBQUSXLCYWF7W4GTTB7NXUFETODPFEVFWYVZFQ6EA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:36.188618657 (84h38m45.911181697s) [noise=true , address=104.36.82.16:4293 , piece_id=S44IV2VXPTE2T2S3UJZXJDXS3KF2P73RVRDK6MKQ7LXHFKY7UDQQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:52528->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188649606 (84h38m45.911212636s) [noise=true , address=104.36.82.13:4186 , piece_id=SJZEBXVKECXJLBRIFQ6XJ2J3YJFXZPBNPRA426DLEALYD2U5ZIQA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188682589 (84h38m45.911245629s) [noise=true , address=104.36.82.4:4802 , piece_id=ETDF7XUCA2IDDPZXPQIEUJMDJSCCLDBB277HC4NB74I5DWRETDKQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:42434->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188688210 (84h38m45.91125125s) [noise=true , address=104.36.82.8:4972 , piece_id=FGHWAMXGXDSFD7Y7NHC3LLEYVOLDGH725HY27RBQR5WC2JSEZ3HA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188693100 (84h38m45.91125614s) [noise=true , address=104.36.82.5:4840 , piece_id=SWGYOU5A43UT2YEOEG2CS2TDHLGB5PCBC663AQH2D75VV4CHY2TQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:42966->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188706796 (84h38m45.911269836s) [seg_pos={0 0}, upload=81234]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 28 00:14:36.418121040 (84h38m46.140684091s) [upload=81234]: Done uploading segments... Jul 28 00:14:36.418151450 (84h38m46.14071449s) [upload=81234]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.814391485s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.231801362s FAIL (2025-07-28 00:14:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== onn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:02.541185363 (84h37m12.263748413s) [seg_pos={0 0}, upload=81217]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:13:02.541379915 (84h37m12.263942965s) [upload=81217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:13:02.541387379 (84h37m12.263950429s) [upload=81217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:13:02.842765569 (84h37m12.565328619s) [upload=81217]: Done uploading segments... Jul 28 00:13:02.842818560 (84h37m12.565381611s) [upload=81217]: Upload finished. err: <nil> Jul 28 00:14:06.832083031 (84h38m16.554646071s) [upload=81234]: Starting upload Jul 28 00:14:06.832143917 (84h38m16.554706957s) [upload=81234]: Done starting upload Jul 28 00:14:06.832192500 (84h38m16.55475554s) [upload=81234]: Uploading segments... Jul 28 00:14:06.832308402 (84h38m16.554871432s) [upload=81234]: Got next segment. Inline: false Jul 28 00:14:06.832332729 (84h38m16.554895769s) [upload=81234]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:14:06.832379038 (84h38m16.554942078s) [upload=81234]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:14:07.069638894 (84h38m16.792201934s) [seg_pos={0 0}, upload=81234]: Begin upload segment... Jul 28 00:14:07.069780064 (84h38m16.792343114s) [seg_pos={0 0}, upload=81234]: Done begin upload segment. Jul 28 00:14:07.069815833 (84h38m16.792378863s) [noise=true , address=45.91.116.14:4191 , piece_id=EWZQOAGO2HJIOHAGPWO4I5KZYMYDH6KEWIWH7OJYZU7X6QX2PJRA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.069823998 (84h38m16.792387028s) [noise=true , address=104.36.82.16:4293 , piece_id=S44IV2VXPTE2T2S3UJZXJDXS3KF2P73RVRDK6MKQ7LXHFKY7UDQQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070154851 (84h38m16.792717891s) [noise=true , address=104.36.82.13:4186 , piece_id=SJZEBXVKECXJLBRIFQ6XJ2J3YJFXZPBNPRA426DLEALYD2U5ZIQA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070406503 (84h38m16.792969543s) [noise=true , address=45.91.116.6:4973 , piece_id=6O6O4FJA7KJ4YCGZZUJFTJTXSGEDC5NGJKP5VD354Z3ATBXGWJJA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.070667613 (84h38m16.793230653s) [noise=true , address=104.36.82.4:4802 , piece_id=ETDF7XUCA2IDDPZXPQIEUJMDJSCCLDBB277HC4NB74I5DWRETDKQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071031479 (84h38m16.793594529s) [noise=true , address=45.91.116.13:4154 , piece_id=MMSBZ6S6KNGT466PK3JDFAYECE2HKAKI2C2XHUHXNMD46HEGDDTQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071287899 (84h38m16.793850939s) [noise=true , address=45.91.116.9:4263 , piece_id=BF625DXSIWXKOS6M52PYDGLXLBGL7PVQV3XJ7XPZVGZ6FPXNQVPQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071502440 (84h38m16.79406548s) [noise=true , address=45.91.116.12:4226 , piece_id=IOROHEXTPBROJPQF7HX7PHV3I4VIH5BGZ7V4MEMZAXHK6TIWJXCA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071694066 (84h38m16.794257106s) [noise=true , address=77.90.31.12:4589 , piece_id=IY2OWMCRRSCLIMNRLRHWZYZMNYHJAKMDJB55CDPZQCRGAIAILDQA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.071994812 (84h38m16.794557862s) [noise=true , address=45.91.116.12:4225 , piece_id=7UQPLI645A4SJWMWEQRBKH2252N4VLJTBGGK3XQ76YCHPE4GGDTQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.072908810 (84h38m16.79547185s) [noise=true , address=77.90.31.16:4735 , piece_id=ZMU734HSUQRZBQUSXLCYWF7W4GTTB7NXUFETODPFEVFWYVZFQ6EA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.072928679 (84h38m16.795491719s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJHTDAEM23LGFK4EK4D3P6S5PMSKHGURLOQ23NR6W6TFCYTDBISA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.073356907 (84h38m16.795919948s) [noise=true , address=77.90.31.10:4518 , piece_id=UQH55LUAWYR5NSVRXXE2LBVJUZQOPRHEDUH2S6E4OY63W5KRYRUQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.073973647 (84h38m16.796536687s) [noise=true , address=104.36.82.8:4972 , piece_id=FGHWAMXGXDSFD7Y7NHC3LLEYVOLDGH725HY27RBQR5WC2JSEZ3HA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.074190893 (84h38m16.796753933s) [noise=true , address=77.90.31.19:4842 , piece_id=6O5K5OL3PZ6DHQG2OTEB3DEJJPJSKPVLX2YPAFKZNJIHRVAZHRZA, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:07.074611729 (84h38m16.797174769s) [noise=true , address=104.36.82.5:4840 , piece_id=SWGYOU5A43UT2YEOEG2CS2TDHLGB5PCBC663AQH2D75VV4CHY2TQ, seg_pos={0 0}, upload=81234]: Uploading piece... Jul 28 00:14:08.828150913 (84h38m18.550713954s) [upload=81234]: Next returned nil segment Jul 28 00:14:08.828193245 (84h38m18.550756285s) [upload=81234]: Waiting for error group managing segments... Jul 28 00:14:09.105320036 (84h38m18.827883076s) [noise=true , address=45.91.116.9:4263 , piece_id=BF625DXSIWXKOS6M52PYDGLXLBGL7PVQV3XJ7XPZVGZ6FPXNQVPQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.106907042 (84h38m18.829470082s) [noise=true , address=77.90.31.12:4589 , piece_id=IY2OWMCRRSCLIMNRLRHWZYZMNYHJAKMDJB55CDPZQCRGAIAILDQA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.149925576 (84h38m18.872488617s) [noise=true , address=45.91.116.13:4154 , piece_id=MMSBZ6S6KNGT466PK3JDFAYECE2HKAKI2C2XHUHXNMD46HEGDDTQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.161168177 (84h38m18.883731237s) [noise=true , address=45.91.116.14:4191 , piece_id=EWZQOAGO2HJIOHAGPWO4I5KZYMYDH6KEWIWH7OJYZU7X6QX2PJRA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.201783955 (84h38m18.924346995s) [noise=true , address=77.90.31.10:4518 , piece_id=UQH55LUAWYR5NSVRXXE2LBVJUZQOPRHEDUH2S6E4OY63W5KRYRUQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.206134097 (84h38m18.928697137s) [noise=true , address=45.91.116.12:4226 , piece_id=IOROHEXTPBROJPQF7HX7PHV3I4VIH5BGZ7V4MEMZAXHK6TIWJXCA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.211591940 (84h38m18.93415498s) [noise=true , address=77.90.31.19:4842 , piece_id=6O5K5OL3PZ6DHQG2OTEB3DEJJPJSKPVLX2YPAFKZNJIHRVAZHRZA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.214527045 (84h38m18.937090085s) [noise=true , address=45.91.116.6:4973 , piece_id=6O6O4FJA7KJ4YCGZZUJFTJTXSGEDC5NGJKP5VD354Z3ATBXGWJJA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.243601201 (84h38m18.966164241s) [noise=true , address=45.91.116.12:4225 , piece_id=7UQPLI645A4SJWMWEQRBKH2252N4VLJTBGGK3XQ76YCHPE4GGDTQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.273963521 (84h38m18.996526561s) [noise=true , address=45.91.116.10:4083 , piece_id=ZJHTDAEM23LGFK4EK4D3P6S5PMSKHGURLOQ23NR6W6TFCYTDBISA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:09.362915428 (84h38m19.085478468s) [noise=true , address=77.90.31.16:4735 , piece_id=ZMU734HSUQRZBQUSXLCYWF7W4GTTB7NXUFETODPFEVFWYVZFQ6EA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: <nil> Jul 28 00:14:36.188618657 (84h38m45.911181697s) [noise=true , address=104.36.82.16:4293 , piece_id=S44IV2VXPTE2T2S3UJZXJDXS3KF2P73RVRDK6MKQ7LXHFKY7UDQQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:52528->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188649606 (84h38m45.911212636s) [noise=true , address=104.36.82.13:4186 , piece_id=SJZEBXVKECXJLBRIFQ6XJ2J3YJFXZPBNPRA426DLEALYD2U5ZIQA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188682589 (84h38m45.911245629s) [noise=true , address=104.36.82.4:4802 , piece_id=ETDF7XUCA2IDDPZXPQIEUJMDJSCCLDBB277HC4NB74I5DWRETDKQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 188.166.225.170:42434->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188688210 (84h38m45.91125125s) [noise=true , address=104.36.82.8:4972 , piece_id=FGHWAMXGXDSFD7Y7NHC3LLEYVOLDGH725HY27RBQR5WC2JSEZ3HA, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188693100 (84h38m45.91125614s) [noise=true , address=104.36.82.5:4840 , piece_id=SWGYOU5A43UT2YEOEG2CS2TDHLGB5PCBC663AQH2D75VV4CHY2TQ, seg_pos={0 0}, upload=81234]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:42966->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:14:36.188706796 (84h38m45.911269836s) [seg_pos={0 0}, upload=81234]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 28 00:14:36.418121040 (84h38m46.140684091s) [upload=81234]: Done uploading segments... Jul 28 00:14:36.418151450 (84h38m46.14071449s) [upload=81234]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.231824124s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.724254749s FAIL (2025-07-28 00:14:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72428099s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X.......................................X............X.....] duration: 1.986556524s FAIL (2025-07-28 00:14:29 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.986582274s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................................X............X....] duration: 2.062455307s FAIL (2025-07-28 00:14:05 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.062481097s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................................................X...XXXXX] duration: 1m0.233484954s FAIL (2025-07-28 00:13:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ) [noise=true , address=104.36.82.55:4210 , piece_id=SJGNL4EHXHVYOCBQJ7IKFKTFD7VCLV4AXC7RKBTKIUU3YJUYG5SQ, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.859711355 (84h36m54.582274395s) [noise=true , address=104.36.82.5:4859 , piece_id=ZRQ5JZUGPOA6VVWCU4SBQZFFTVDSUSIQZ7ZMMQ6JEBGMBDOK5FUQ, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.859923941 (84h36m54.582486981s) [noise=true , address=104.36.82.55:4220 , piece_id=MZIPRJLZCQIQVQKBXZKZGDEJUBGGMRDYZIMHKYFFHC7UARABVSUQ, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.861356453 (84h36m54.583919493s) [noise=true , address=104.36.82.42:4813 , piece_id=BDVUH5V5VONX3XOVGQAL2YCJOGA3SU77UVOLPAX7MQ7U5I22ZHVA, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.861593386 (84h36m54.584156426s) [noise=true , address=104.36.82.13:4211 , piece_id=LEMBL5KIESNZ756A4SLLWOGBR2MPINI5ARTEOW5A53VCJG2RNPEA, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.861849255 (84h36m54.584412295s) [noise=true , address=79.127.220.98:4404 , piece_id=FQ2WSKKLXKMLK6YWROLW6IQN7LPQRDFA7JQK6IXBPPG5AF5HZGLQ, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:44.862108752 (84h36m54.584671792s) [noise=true , address=79.127.201.215:4359 , piece_id=YLHIPOQIJ54N63YRYWWCQOK7KUCY6NGXXPCMQAQKXNODW6M3NQYQ, seg_pos={0 0}, upload=81215]: Uploading piece... Jul 28 00:12:46.214784411 (84h36m55.937347451s) [upload=81215]: Next returned nil segment Jul 28 00:12:46.214802365 (84h36m55.937365395s) [upload=81215]: Waiting for error group managing segments... Jul 28 00:12:46.716382710 (84h36m56.43894575s) [noise=true , address=79.127.220.98:4404 , piece_id=FQ2WSKKLXKMLK6YWROLW6IQN7LPQRDFA7JQK6IXBPPG5AF5HZGLQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: <nil> Jul 28 00:12:46.805452172 (84h36m56.528015213s) [noise=true , address=79.127.201.215:4359 , piece_id=YLHIPOQIJ54N63YRYWWCQOK7KUCY6NGXXPCMQAQKXNODW6M3NQYQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: <nil> Jul 28 00:13:43.766587954 (84h37m53.489150994s) [noise=true , address=104.36.82.50:4020 , piece_id=ZC2XYMUQA5BF5BRWJP5NVR6U4ZONOQTYMQ45ETFTKTLMTDHDSF3A, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:129LAxYoAkRhCwZLWqeTHSUUMnAQZTotqKyBr3vMTDKF9cEKNCk, address:104.36.82.50:4020): protocol: write tcp 188.166.225.170:47826->104.36.82.50:4020: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766643010 (84h37m53.48920605s) [noise=true , address=104.36.82.52:4085 , piece_id=DV5YPK5W7E6BEGJPHZXBNLATNHQRUXGWQIF4RRY4JTQ7PNPCTKKA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:48572->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766679439 (84h37m53.489242479s) [noise=true , address=104.36.82.5:4859 , piece_id=ZRQ5JZUGPOA6VVWCU4SBQZFFTVDSUSIQZ7ZMMQ6JEBGMBDOK5FUQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: rpc: dial tcp 104.36.82.5:4859: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766712322 (84h37m53.489275362s) [noise=true , address=104.36.82.3:4771 , piece_id=MJHFXHKQCSZDJC2DQU2TRNFEN5ADFDFKJYBCSGAUHCCH7JLCZCDA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1rcgrMxg2fRfvh7EA1eUehzD5rSH1krUd5W5eC8KTo7vjYWFLL, address:104.36.82.3:4771): protocol: write tcp 188.166.225.170:44058->104.36.82.3:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766745386 (84h37m53.489308426s) [noise=true , address=104.36.82.41:4775 , piece_id=Q2NX6BXPDITSCH5XSAVVHHLWOZYF3W2PG2PBLEEPMF6M5JBCMTBQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:166yRbNnnDeZRifM5q7vUVLx7iV5cZ9GqQnmvD9AbUUJsaCjmj, address:104.36.82.41:4775): protocol: write tcp 188.166.225.170:43222->104.36.82.41:4775: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766774491 (84h37m53.489337532s) [noise=true , address=104.36.82.14:4231 , piece_id=JUOGPDT2TDR5JXC5YSQGEE4TOXY45N6NI6T55JCEUAD4LMPQMEUQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:104.36.82.14:4231): protocol: write tcp 188.166.225.170:60514->104.36.82.14:4231: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766805651 (84h37m53.489368691s) [noise=true , address=104.36.82.11:4140 , piece_id=5I6VSDKFGANPEZVJMOWNZJNQLZKOZPKRGTKCYANBLPESMDX5MPRA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: write tcp 188.166.225.170:51798->104.36.82.11:4140: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766840497 (84h37m53.489403537s) [noise=true , address=104.36.82.55:4210 , piece_id=SJGNL4EHXHVYOCBQJ7IKFKTFD7VCLV4AXC7RKBTKIUU3YJUYG5SQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: write tcp 188.166.225.170:37270->104.36.82.55:4210: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766853022 (84h37m53.489416062s) [noise=true , address=104.36.82.53:4154 , piece_id=QJ4VPXO6VMZ45LUZPWOSIISYLKLGOC6AFWOS4IPBVUNI354N4KXA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 188.166.225.170:46258->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766883069 (84h37m53.489446109s) [noise=true , address=104.36.82.55:4220 , piece_id=MZIPRJLZCQIQVQKBXZKZGDEJUBGGMRDYZIMHKYFFHC7UARABVSUQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: rpc: dial tcp 104.36.82.55:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766913137 (84h37m53.489476207s) [noise=true , address=104.36.82.40:4975 , piece_id=4HKDNVUYF3TDPFBLOL2256EHNCTN4WXMNWEC5GCMAVVMC4UTQCWQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1fUjYWso7fh5NoXn8jc9Pu8aukkP2PbPdtTKChfaQ8GiWtn1RV, address:104.36.82.40:4975): protocol: write tcp 188.166.225.170:58002->104.36.82.40:4975: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766922745 (84h37m53.489485775s) [noise=true , address=104.36.82.42:4813 , piece_id=BDVUH5V5VONX3XOVGQAL2YCJOGA3SU77UVOLPAX7MQ7U5I22ZHVA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: write tcp 188.166.225.170:44210->104.36.82.42:4813: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766959525 (84h37m53.489522555s) [noise=true , address=104.36.82.13:4211 , piece_id=LEMBL5KIESNZ756A4SLLWOGBR2MPINI5ARTEOW5A53VCJG2RNPEA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: rpc: dial tcp 104.36.82.13:4211: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766964535 (84h37m53.489527605s) [noise=true , address=104.36.82.20:4450 , piece_id=NNRJMRKMGL2EPDR7ZD4A3PLP2Y67HQNWIRJLKQAUEW3HYMQHC2QQ, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:12UeL1LySKQySa8wjvG4FBs69cE1iWkW4Diz83nu1PxBCKxyvki, address:104.36.82.20:4450): protocol: write tcp 188.166.225.170:37384->104.36.82.20:4450: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.766989111 (84h37m53.489552151s) [noise=true , address=104.36.82.54:4187 , piece_id=TWBFDL62I7CVNI7TDSCAH66FWN5JKILAGSJ54EIZDVSZFM5KPTTA, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 188.166.225.170:48086->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.767001465 (84h37m53.489564506s) [noise=true , address=104.36.82.44:4904 , piece_id=DMZ7GELFROXJUYLMML4ABUUWDWGTWKIDWEMLHEKM6FSKOAOBYQ5A, seg_pos={0 0}, upload=81215]: Done uploading piece. err: ecclient: upload failed (node:1dC7mC2Cbxmys7ukEkELyF9Us7F1qtptSP6yDJh1EfjVJ6uqsC, address:104.36.82.44:4904): protocol: write tcp 188.166.225.170:33020->104.36.82.44:4904: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:13:43.767033296 (84h37m53.489596326s) [seg_pos={0 0}, upload=81215]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:13:43.998323681 (84h37m53.720886731s) [upload=81215]: Done uploading segments... Jul 28 00:13:43.998357275 (84h37m53.720920316s) [upload=81215]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.233507256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.749913645s FAIL (2025-07-28 00:13:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.749940466s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X.......................................X............X...] duration: 2.208255268s FAIL (2025-07-28 00:13:05 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 2.208286136s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [....................................................X...XXXX] duration: 1m0.24835735s FAIL (2025-07-28 00:12:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== , piece_id=UYBNJDRX3EXIEKH4GF2LXSWH5QD7LHZFYOPNAQZTL3IL426DF7YQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.325662567 (84h35m54.048225607s) [noise=true , address=104.36.82.13:4189 , piece_id=CAUOLXHDX2CWR5NDR53RHHWIQ6LH6QDVOPRN4EGSAM37UDPLUTZQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.326017496 (84h35m54.048580536s) [noise=true , address=104.36.82.51:4074 , piece_id=TQ7QAKXMAXOGRF46IKAMZ3VA6IWBJCTB6AFTUZRAG3CQ74HX5OUQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.326901968 (84h35m54.049465008s) [noise=true , address=79.127.201.215:4352 , piece_id=6KL7CJ47H6J72SCSBC5DDDLXHDSPNJ4W43XTBMBK2DW3COUC6NDQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.327571920 (84h35m54.050134961s) [noise=true , address=104.36.82.22:4533 , piece_id=TEAHFVPSGR6JASJLV6YZPVCS7EDL7YA5N2ZKUAGBKRELNVHQOQUA, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.328156308 (84h35m54.050719368s) [noise=true , address=104.36.82.10:4107 , piece_id=XPLJGM3XZVA7W2R4FJ6BF2DGQKZ653M3P7WWHZ6RSO5NLIRVHLUA, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.328822813 (84h35m54.051385863s) [noise=true , address=104.36.82.5:4859 , piece_id=DFNXPHR367TJQ54B2QHOEDCF6KL2NM7GMME5JFHHR7EC46IBHQPQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.329321818 (84h35m54.051884858s) [noise=true , address=104.36.82.50:4016 , piece_id=3TSXGIYBM4BGBK7MY6GWTLVVCFV4RBUDXLEPFJWS6UZB77FYDADQ, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:44.329762912 (84h35m54.052325952s) [noise=true , address=79.127.205.235:4682 , piece_id=3D56MFGYTYBYU5IO3YJZNH4CI3776SGLJY24XUE4V2YQLDKR2SMA, seg_pos={0 0}, upload=81199]: Uploading piece... Jul 28 00:11:45.435025990 (84h35m55.15758903s) [upload=81199]: Next returned nil segment Jul 28 00:11:45.435040607 (84h35m55.157603647s) [upload=81199]: Waiting for error group managing segments... Jul 28 00:11:45.714476126 (84h35m55.437039156s) [noise=true , address=79.127.205.235:4682 , piece_id=3D56MFGYTYBYU5IO3YJZNH4CI3776SGLJY24XUE4V2YQLDKR2SMA, seg_pos={0 0}, upload=81199]: Done uploading piece. err: <nil> Jul 28 00:11:46.325136137 (84h35m56.047699167s) [noise=true , address=79.127.201.215:4352 , piece_id=6KL7CJ47H6J72SCSBC5DDDLXHDSPNJ4W43XTBMBK2DW3COUC6NDQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: <nil> Jul 28 00:12:43.223228696 (84h36m52.945791736s) [noise=true , address=104.36.82.50:4016 , piece_id=3TSXGIYBM4BGBK7MY6GWTLVVCFV4RBUDXLEPFJWS6UZB77FYDADQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1jC98jEWn6K8jFHPA9knJeHAkuNEmcS7ThxUvbySDfbq6FmKVR, address:104.36.82.50:4016): protocol: rpc: dial tcp 104.36.82.50:4016: i/o timeout; conn closed: op Jul 28 00:12:43.223255477 (84h36m52.945818517s) [noise=true , address=104.36.82.10:4107 , piece_id=XPLJGM3XZVA7W2R4FJ6BF2DGQKZ653M3P7WWHZ6RSO5NLIRVHLUA, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:104.36.82.10:4107): protocol: rpc: dial tcp 104.36.82.10:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.223992818 (84h36m52.946555858s) [noise=true , address=104.36.82.55:4195 , piece_id=JGBRIVJF3G2UG6PLD25W7KAC2UPDHAJTF4TXOKR5KWD7LI6DHIRQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224051991 (84h36m52.946615031s) [noise=true , address=104.36.82.51:4074 , piece_id=TQ7QAKXMAXOGRF46IKAMZ3VA6IWBJCTB6AFTUZRAG3CQ74HX5OUQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1DBQpuE1EC1ENCy67We5QhMT6zsa2AbeMDSv8hmQLjokGRvoxc, address:104.36.82.51:4074): protocol: write tcp 188.166.225.170:54962->104.36.82.51:4074: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224118759 (84h36m52.946681799s) [noise=true , address=104.36.82.34:4570 , piece_id=B7SOEIRJVUFC4LAGU35SZZBDWWESX5LSXFZ6NTVRGWAGH764L42A, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:12FX9wGFEYh96uNsKZLGCNRvEGLCmSmcG8a5HxU67yUU6eT1W4f, address:104.36.82.34:4570): protocol: rpc: dial tcp 104.36.82.34:4570: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224163244 (84h36m52.946726284s) [noise=true , address=104.36.82.20:4449 , piece_id=XQVIISPOOAQZKNXM6DFIKHWAARMQ3XGGOMIA4YYVZ3UWL6DWBJBQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1r53UYADNA4XuUzpWdAvUh6T3iUJSbX9YvqnimzoPTgMfSmBWF, address:104.36.82.20:4449): protocol: rpc: dial tcp 104.36.82.20:4449: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224200565 (84h36m52.946763605s) [noise=true , address=104.36.82.52:4088 , piece_id=JZHR6ED3COIONQ535TJD2MLKYNVTIUTU7OKUB5NRQXHLLGI5SZ3A, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: rpc: dial tcp 104.36.82.52:4088: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224250771 (84h36m52.946813811s) [noise=true , address=104.36.82.15:4271 , piece_id=3PSK2BBYK3IM2ENUY73ME4FZBG7VV5E6GSYZH233RCJGL2P5K7MA, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu, address:104.36.82.15:4271): protocol: write tcp 188.166.225.170:59106->104.36.82.15:4271: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224302250 (84h36m52.94686531s) [noise=true , address=104.36.82.5:4854 , piece_id=UYBNJDRX3EXIEKH4GF2LXSWH5QD7LHZFYOPNAQZTL3IL426DF7YQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:19pxQXokdTJ9W6k5aZrzGZSoAn9p1NqpwbTWMSC6U7ncr8ipwy, address:104.36.82.5:4854): protocol: rpc: dial tcp 104.36.82.5:4854: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224785615 (84h36m52.947348675s) [noise=true , address=104.36.82.14:4228 , piece_id=HHOYOGXEJUCVJXBKZZZ64CZUHFJTYAGA33JWT74HQRDHHMUASVWQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: rpc: dial tcp 104.36.82.14:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224860487 (84h36m52.947423527s) [noise=true , address=104.36.82.11:4132 , piece_id=PMZUGZJWWAPNDLMQQJVKJQG5LETQNRXWXTAILVSOY2SASRTAKJWQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:17856PRHooRiPyrBYLJggh5uYpdWA9CnGxgCrmVesToNu7FeLb, address:104.36.82.11:4132): protocol: rpc: dial tcp 104.36.82.11:4132: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224914781 (84h36m52.947477831s) [noise=true , address=104.36.82.53:4131 , piece_id=PL6VHDO66CDQ6LTMOR4HFV6AAHNSH3HTMIR5RITJ5J27VSE2BEWQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1uhyc5ozrF9ndeEnQXtjc9o3tsYiAnW1dzeuHV3kv1JS4T4a8C, address:104.36.82.53:4131): protocol: rpc: dial tcp 104.36.82.53:4131: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224940400 (84h36m52.94750344s) [noise=true , address=104.36.82.22:4533 , piece_id=TEAHFVPSGR6JASJLV6YZPVCS7EDL7YA5N2ZKUAGBKRELNVHQOQUA, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:12LXK5pbXavbPZh4YSuEkgk7wsRqwvcVEHv4w9FDRYqSkEU89VF, address:104.36.82.22:4533): protocol: write tcp 188.166.225.170:33168->104.36.82.22:4533: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224966340 (84h36m52.94752937s) [noise=true , address=104.36.82.42:4811 , piece_id=SUIJYI3BQCUO4NVXX5FE2ML76652WXILAOPVPHOEJUEMUWAMEETA, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: write tcp 188.166.225.170:46778->104.36.82.42:4811: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.224977803 (84h36m52.947540853s) [noise=true , address=104.36.82.13:4189 , piece_id=CAUOLXHDX2CWR5NDR53RHHWIQ6LH6QDVOPRN4EGSAM37UDPLUTZQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:104.36.82.13:4189): protocol: rpc: dial tcp 104.36.82.13:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.225009012 (84h36m52.947572062s) [noise=true , address=104.36.82.5:4859 , piece_id=DFNXPHR367TJQ54B2QHOEDCF6KL2NM7GMME5JFHHR7EC46IBHQPQ, seg_pos={0 0}, upload=81199]: Done uploading piece. err: ecclient: upload failed (node:1EiYA6gHiLyHsMfPevj2gLqmqttu1knLw3AM8SbWWtgKEf7Csu, address:104.36.82.5:4859): protocol: write tcp 188.166.225.170:43948->104.36.82.5:4859: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:12:43.225033569 (84h36m52.947596609s) [seg_pos={0 0}, upload=81199]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:12:43.469862617 (84h36m53.192425657s) [upload=81199]: Done uploading segments... Jul 28 00:12:43.469886983 (84h36m53.192450023s) [upload=81199]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.248382177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X.......................................X............X..] duration: 1.881094612s FAIL (2025-07-28 00:12:38 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.881110813s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.904795418s FAIL (2025-07-28 00:12:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904819283s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......................................X............X.] duration: 1.824028299s FAIL (2025-07-28 00:12:07 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 1.824063026s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.....................................................X...XXX] duration: 1m0.233579818s FAIL (2025-07-28 00:11:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 4) ======LOGS====== s) [noise=true , address=104.36.82.41:4766 , piece_id=FM4Y6NODBVPQYENOPPSROIVNPRAUERO5JGWKQ36BONSA3TUA7YMQ, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.884066576 (84h34m53.606629616s) [noise=true , address=104.36.82.8:4994 , piece_id=RNZBGEYTK5Z4WS3LUH32F4IV4L3QOWLHQCM36TSEBG25N2RRLYNQ, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.884284081 (84h34m53.606847121s) [noise=true , address=104.36.82.45:4908 , piece_id=5SJDQVNARCLWUTTHM5TPRB3KUZYE3YGVIFFKFY7NXNIIR4SDTIEQ, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.884535182 (84h34m53.607098222s) [noise=true , address=104.36.82.14:4247 , piece_id=CWNR4OU22ZNEK4GKOYL7QJW3WPASM6E26DEGSZC4ER54OGSP23ZA, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.884729384 (84h34m53.607292424s) [noise=true , address=104.36.82.9:4039 , piece_id=COAKKDTPOMU2XY3C3JV4WKKKN2RYO6OSOE75FFZXUPTL5WYWDEQQ, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.885157613 (84h34m53.607720653s) [noise=true , address=64.49.13.138:4482 , piece_id=U3RCE7OUPTYF2O7NJY7QIMP3LKI7MZ3SRTGCDWRZPDZZY4XRGI5A, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.885436476 (84h34m53.607999516s) [noise=true , address=104.36.82.4:4812 , piece_id=VERYPZSXYU532V2UIS22XNCYJLRZBWJJA6E4BQ6LWO23DTOJNUCA, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.885507883 (84h34m53.608070923s) [noise=true , address=104.36.82.13:4189 , piece_id=MUXAJN726ZBDUXY73H5U7VOSZM56YQHJMFAJKRDSFSFIVRNZSXDA, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:43.885733224 (84h34m53.608296254s) [noise=true , address=77.90.31.16:4744 , piece_id=VUBTUBVZLUCJNLTWMV5KJUKRXD2CZNMTTICI7NYPVAIRF425HQRA, seg_pos={0 0}, upload=81182]: Uploading piece... Jul 28 00:10:44.989208530 (84h34m54.71177157s) [upload=81182]: Next returned nil segment Jul 28 00:10:44.989226776 (84h34m54.711789816s) [upload=81182]: Waiting for error group managing segments... Jul 28 00:10:45.638422936 (84h34m55.360985966s) [noise=true , address=79.127.205.245:4246 , piece_id=KEEASWA4WSS34D3GQG4N3CHB3PDWFBACM5MIM2QJJG6EAKN6WVFQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: <nil> Jul 28 00:10:45.761990220 (84h34m55.48455326s) [noise=true , address=64.49.13.138:4482 , piece_id=U3RCE7OUPTYF2O7NJY7QIMP3LKI7MZ3SRTGCDWRZPDZZY4XRGI5A, seg_pos={0 0}, upload=81182]: Done uploading piece. err: <nil> Jul 28 00:10:45.773216609 (84h34m55.495779649s) [noise=true , address=77.90.31.16:4744 , piece_id=VUBTUBVZLUCJNLTWMV5KJUKRXD2CZNMTTICI7NYPVAIRF425HQRA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: <nil> Jul 28 00:10:45.978370629 (84h34m55.700933659s) [noise=true , address=104.36.82.7:4923 , piece_id=4MCWYW5CYHUTDBSVPRR2PQRW3QG2D4VG6VTX3M3WV5O25LJY7ZXA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: <nil> Jul 28 00:11:42.764442585 (84h35m52.487005635s) [noise=true , address=104.36.82.13:4189 , piece_id=MUXAJN726ZBDUXY73H5U7VOSZM56YQHJMFAJKRDSFSFIVRNZSXDA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1jH3757i2F3q18XSpoXznsysZPaXmEnqynpLPDupgRfxJrTgKv, address:104.36.82.13:4189): protocol: write tcp 188.166.225.170:36278->104.36.82.13:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764587282 (84h35m52.487150333s) [noise=true , address=104.36.82.53:4135 , piece_id=FPRYUQI3P7WSIMQXJDO6WF6OH7Z5N6HUKVSLW2GNEJOFOBLOX67Q, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:128j2QoFq4UDWCXjV8uChmN9KB1sQMnhdF6hGgNQygdqHvw9tPj, address:104.36.82.53:4135): protocol: write tcp 188.166.225.170:39796->104.36.82.53:4135: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764637449 (84h35m52.487200489s) [noise=true , address=104.36.82.41:4766 , piece_id=FM4Y6NODBVPQYENOPPSROIVNPRAUERO5JGWKQ36BONSA3TUA7YMQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1k6zdamuKPkBdzyNZNWx4FCmaTkv33PchcwvTRR8bgcETBs4K7, address:104.36.82.41:4766): protocol: write tcp 188.166.225.170:60174->104.36.82.41:4766: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764648950 (84h35m52.48721199s) [noise=true , address=104.36.82.54:4181 , piece_id=DDWGCZQT2SKPR6ITPJTKARNRQKCHRDSOK4SPEX7KE5OLXQSIHUIA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:12gGf5E7tzF2TXtLWgpK55t24SC1BpTSeC9G66nPmR1ARnDrAE8, address:104.36.82.54:4181): protocol: write tcp 188.166.225.170:50350->104.36.82.54:4181: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764703134 (84h35m52.487266174s) [noise=true , address=104.36.82.51:4049 , piece_id=KRDFFZ5QQD4WRD6QCNMNDEVK3R55LEMJ5GYSS6YJ56ORCD5QHYNQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: write tcp 188.166.225.170:50296->104.36.82.51:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764732821 (84h35m52.487295861s) [noise=true , address=104.36.82.9:4039 , piece_id=COAKKDTPOMU2XY3C3JV4WKKKN2RYO6OSOE75FFZXUPTL5WYWDEQQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:12P11WVW9mVXw5ra2gZ3Rezon6wZi8GrN5CchQN3MiVU4ts3L6V, address:104.36.82.9:4039): protocol: write tcp 188.166.225.170:54378->104.36.82.9:4039: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764756025 (84h35m52.487319075s) [noise=true , address=104.36.82.8:4994 , piece_id=RNZBGEYTK5Z4WS3LUH32F4IV4L3QOWLHQCM36TSEBG25N2RRLYNQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:13nneaPpsRT7mVEc6cDt7GiZSMTU68Nnk1rr9Lbn5ycZkmQGrB, address:104.36.82.8:4994): protocol: write tcp 188.166.225.170:37814->104.36.82.8:4994: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764805290 (84h35m52.48736833s) [noise=true , address=104.36.82.44:4900 , piece_id=2BTSPXQAY7343QCS2TQSGYM5R2IBSOYISA5BJS5GAESQZTGHXR3Q, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1R8K65zsCspu7W2oAnvtpz7zEsQFkDG1e5BAp6faQYMQrkh4oQ, address:104.36.82.44:4900): protocol: write tcp 188.166.225.170:49444->104.36.82.44:4900: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764853161 (84h35m52.487416201s) [noise=true , address=104.36.82.20:4444 , piece_id=PSHLTCBAJDJ552A6Q4OQAS3DJMOFEOHC7HPSJNLKFH5VJZKRDPZQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:12m5PditUsvMEejqnuEBZ6jLunbkTw9dEkmGzLcBg4y2XAwhBTD, address:104.36.82.20:4444): protocol: write tcp 188.166.225.170:48144->104.36.82.20:4444: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764883088 (84h35m52.487446128s) [noise=true , address=104.36.82.15:4284 , piece_id=TFS46DLIKXFOQPBRDZHP6JQSHG6CZRL5RX7ZR4P66RPOUBNANNHA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1YAnLu8onh19Mx524VikQzCHEFW4U8dgtuLx8amtQAiReSCisF, address:104.36.82.15:4284): protocol: write tcp 188.166.225.170:44114->104.36.82.15:4284: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764904188 (84h35m52.487467228s) [noise=true , address=104.36.82.3:4789 , piece_id=AYGT74A5RXC3ZZY2TQUG2RBC43A3BSBR355CLX6GWU452IDDU6YA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:12eQnnZm3aoAEA6cHJshERFmZuyh5cUjqzYK687CS27CQ8nGzrT, address:104.36.82.3:4789): protocol: write tcp 188.166.225.170:37356->104.36.82.3:4789: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764933184 (84h35m52.487496244s) [noise=true , address=104.36.82.14:4247 , piece_id=CWNR4OU22ZNEK4GKOYL7QJW3WPASM6E26DEGSZC4ER54OGSP23ZA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:104.36.82.14:4247): protocol: write tcp 188.166.225.170:54402->104.36.82.14:4247: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764942492 (84h35m52.487505522s) [noise=true , address=104.36.82.45:4908 , piece_id=5SJDQVNARCLWUTTHM5TPRB3KUZYE3YGVIFFKFY7NXNIIR4SDTIEQ, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:1Qm73bcHkYSxHw4GGgnB8u1MPpCiQTVm9D35DFYrhrQf49M2y2, address:104.36.82.45:4908): protocol: write tcp 188.166.225.170:33678->104.36.82.45:4908: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764953553 (84h35m52.487516593s) [noise=true , address=104.36.82.4:4812 , piece_id=VERYPZSXYU532V2UIS22XNCYJLRZBWJJA6E4BQ6LWO23DTOJNUCA, seg_pos={0 0}, upload=81182]: Done uploading piece. err: ecclient: upload failed (node:12gTRhEnzLoKmQYRttr4hwiTbJ4y3yqMUS1Qg7F2yohb81sd5KV, address:104.36.82.4:4812): protocol: write tcp 188.166.225.170:59874->104.36.82.4:4812: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:11:42.764974523 (84h35m52.487537553s) [seg_pos={0 0}, upload=81182]: Done waiting for segment. successful: 4 optimal: 14 errs: <nil> Jul 28 00:11:42.995571571 (84h35m52.718134641s) [upload=81182]: Done uploading segments... Jul 28 00:11:42.995600947 (84h35m52.718163987s) [upload=81182]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 4) Duration: 1m0.233592382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.87103676s FAIL (2025-07-28 00:11:35 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.871068119s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)