Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.2312821s FAIL (2025-07-28 01:15:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== UYPV5IBKG7U2WW74TQ5QO33W5BAIQ, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:45.115168157 (85h38m54.837731197s) [noise=true , address=104.36.82.51:4084 , piece_id=6LA252S44EWYIQIUPWGSEEP2E4OGYLZEZ5UQMDGFOIR6IYV6RVWA, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:45.115479063 (85h38m54.838042103s) [noise=true , address=104.36.82.53:4146 , piece_id=AAGKTK7DUXJ2LAQ74ENJUXBJX6IA7BPA4PQ5MWM63VVZTOBHK3DQ, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:45.115849732 (85h38m54.838412772s) [noise=true , address=104.36.82.5:4849 , piece_id=VINA7OMUZDS6GHBG7JMNC4JRUR5WWWNQDYHWEBJHQTFG7BSVZZDQ, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:45.116277671 (85h38m54.838840701s) [noise=true , address=104.36.82.8:4999 , piece_id=I3KKRUTBFAO6WAVUVMX6DOXVILSMWYNVQQHJ5UB5P2AAPNNYWMQQ, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:45.116678978 (85h38m54.839242018s) [noise=true , address=104.36.82.11:4117 , piece_id=V7F4S7DZOU2ZFI5KT6LDQBRLST6FZUSSQCFJFZCGOXKJOHI44MJQ, seg_pos={0 0}, upload=82160]: Uploading piece... Jul 28 01:14:46.601737361 (85h38m56.324300401s) [upload=82160]: Next returned nil segment Jul 28 01:14:46.601757660 (85h38m56.32432071s) [upload=82160]: Waiting for error group managing segments... Jul 28 01:14:49.005545590 (85h38m58.72810864s) [noise=true , address=79.127.201.215:4351 , piece_id=SXU7LOBSRNK6YALIZGIK2CDBY3FTTSPMIFWT6GS7JQ7WLWFHFSNQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: <nil> Jul 28 01:15:44.020751249 (85h39m53.743314289s) [noise=true , address=104.36.82.46:4511 , piece_id=KBR6LGFSJBB3S5LXJAO77GCZU3XYDUSG7F3T5SJXQTQRJDZED23A, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12Hbb5bsmU7eU4ejMtupHJByLcPHgEAUNiTbgh3rqmgVRVCfdh6, address:104.36.82.46:4511): protocol: write tcp 188.166.225.170:48896->104.36.82.46:4511: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020795423 (85h39m53.743358463s) [noise=true , address=104.36.82.6:4903 , piece_id=GR4W26VDRGFPPIITJQFGBLFWOR3VSFXFTNFDMGBJ6SPLOILDIMDQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1zFBuDaxAtbFho2wumoXWSoQbB7y9VaSFizz31gV8n1fLK6syE, address:104.36.82.6:4903): protocol: write tcp 188.166.225.170:50286->104.36.82.6:4903: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020825140 (85h39m53.74338818s) [noise=true , address=104.36.82.56:4257 , piece_id=4SB5KEKFW5LIU3AVUSGAPZYTVSZ4ZL3LEDG4MJBOYW35ANCXC4BA, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12iw1KEPBAYhvSachvkigv7bPXfGN5XEugvxGC5YTBV2Muymir3, address:104.36.82.56:4257): protocol: write tcp 188.166.225.170:59078->104.36.82.56:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020856369 (85h39m53.743419409s) [noise=true , address=104.36.82.54:4186 , piece_id=FKNUA7VIORXP4PE2C7EIV434TXSX6CVVJU42NHOGN5L7G23S5N6Q, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12Gu3vXy2JUkKDaF4MkMCgVmYspW4Ez8EAfv6e4GeCkrwYTe5T, address:104.36.82.54:4186): protocol: write tcp 188.166.225.170:42802->104.36.82.54:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020887410 (85h39m53.74345045s) [noise=true , address=104.36.82.41:4787 , piece_id=VCCFYMGQXQ5AN4G32T25TFKKCQ76QMB2JBFUZRWAJR55F7Z7W3IA, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12aKqg1aNsv8Yi7fE4asGz8bijcPewWDvUukKcf5SFnzgARju43, address:104.36.82.41:4787): protocol: write tcp 188.166.225.170:42370->104.36.82.41:4787: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020920643 (85h39m53.743483683s) [noise=true , address=104.36.82.55:4219 , piece_id=SSVWEK67LB7LCQT75CDN7WZRGOV4RR7BCHBQECKRKXKY436EQNGA, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12TqSxiG3LYYc3W5T6wqiuuKwoh1LvCdntou7ve6TAEbXYf2QKn, address:104.36.82.55:4219): protocol: write tcp 188.166.225.170:39470->104.36.82.55:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020949478 (85h39m53.743512518s) [noise=true , address=104.36.82.12:4170 , piece_id=5XW2DRY2J3P2TAYCZ54ISQFDQH7TSJSPMCEQ6NOFLHUTTGJPC6HQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1EfVDsPfsWymV1yfufQw2BidpCGqqqsmtrvoid67a5ug9YYyyd, address:104.36.82.12:4170): protocol: write tcp 188.166.225.170:36752->104.36.82.12:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.020977171 (85h39m53.743540211s) [noise=true , address=104.36.82.13:4194 , piece_id=ZSRVZ3C6VKKNVSSYXU5O5QXHAQWG5D6TPWDUPAS4UFJTDCBXLPSQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: write tcp 188.166.225.170:49838->104.36.82.13:4194: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021003782 (85h39m53.743566822s) [noise=true , address=104.36.82.46:4482 , piece_id=ED2BR4NQNUME6QEOGF3NAYEAQJKDWOPGUQQQXMJ4MNQH3C2SE23Q, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: write tcp 188.166.225.170:36822->104.36.82.46:4482: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021032126 (85h39m53.743595166s) [noise=true , address=104.36.82.8:4999 , piece_id=I3KKRUTBFAO6WAVUVMX6DOXVILSMWYNVQQHJ5UB5P2AAPNNYWMQQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12Y993kGnB4JusXCcZ4KGyGpRBwoHksVBPzK77ePMw9G7djQEyY, address:104.36.82.8:4999): protocol: write tcp 188.166.225.170:38464->104.36.82.8:4999: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021062875 (85h39m53.743625915s) [noise=true , address=104.36.82.11:4117 , piece_id=V7F4S7DZOU2ZFI5KT6LDQBRLST6FZUSSQCFJFZCGOXKJOHI44MJQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:122Htg72f5eUUutf1p3fYQpy1Wkf6AgdaQV83HHYye2n8zm3the, address:104.36.82.11:4117): protocol: write tcp 188.166.225.170:52576->104.36.82.11:4117: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021093934 (85h39m53.743656974s) [noise=true , address=104.36.82.7:4927 , piece_id=QRIUIFUNAYY57RUIQEKF36RJZE77NVYPLNLCSO24JFUUILMXQ7TQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1AD5Ju87nY6LL5aTbFFsfNJRt8gkyGExsSrJLf6AP2q1nd9HBr, address:104.36.82.7:4927): protocol: write tcp 188.166.225.170:53678->104.36.82.7:4927: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021121677 (85h39m53.743684717s) [noise=true , address=104.36.82.51:4084 , piece_id=6LA252S44EWYIQIUPWGSEEP2E4OGYLZEZ5UQMDGFOIR6IYV6RVWA, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:121BaY9pfRq2a2sZTqmPA3NrXNh9pnckJj21N13sV4jhkuo2gP9, address:104.36.82.51:4084): protocol: write tcp 188.166.225.170:37810->104.36.82.51:4084: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021149971 (85h39m53.743713011s) [noise=true , address=104.36.82.53:4146 , piece_id=AAGKTK7DUXJ2LAQ74ENJUXBJX6IA7BPA4PQ5MWM63VVZTOBHK3DQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1ecwZFchyb5HmyKwRFfLDaVGck3xDWmgrVNYadHoTLWY5Z2Qs1, address:104.36.82.53:4146): protocol: write tcp 188.166.225.170:59986->104.36.82.53:4146: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021178285 (85h39m53.743741326s) [noise=true , address=104.36.82.54:4170 , piece_id=H44KISQEVI4QX5R2DCSGLY6UZKCAERJ6NYTGK6WUOEUCQ3GAYFKA, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 188.166.225.170:59878->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021206750 (85h39m53.74376979s) [noise=true , address=104.36.82.10:4089 , piece_id=65X3IGE3GFSGNMNDKKQBAQJUYPV5IBKG7U2WW74TQ5QO33W5BAIQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:40978->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:15:44.021304147 (85h39m53.743867177s) [noise=true , address=104.36.82.5:4849 , piece_id=VINA7OMUZDS6GHBG7JMNC4JRUR5WWWNQDYHWEBJHQTFG7BSVZZDQ, seg_pos={0 0}, upload=82160]: Done uploading piece. err: ecclient: upload failed (node:1n2gSzVmNCEZcfVsFKibjXHs2jMr82D3SWQEykFQQ4uyhhnrj1, address:104.36.82.5:4849): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:15:44.021318494 (85h39m53.743881534s) [seg_pos={0 0}, upload=82160]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:15:44.250103085 (85h39m53.972666125s) [upload=82160]: Done uploading segments... Jul 28 01:15:44.250128343 (85h39m53.972691383s) [upload=82160]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.231302138s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X.......X.....X...............................X.....XXX.....] duration: 6.104109579s FAIL (2025-07-28 01:15:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.10413125s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X...................X..................X.........] duration: 5.922344216s FAIL (2025-07-28 01:14:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.922369555s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X.......X.....X...............................X.....XXX....] duration: 5.796735994s FAIL (2025-07-28 01:14:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.796761763s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244876856s FAIL (2025-07-28 01:14:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== SW2YP2J5LOY2IPA, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.682894888 (85h37m54.405457928s) [noise=true , address=104.36.82.49:4987 , piece_id=JKIKFUXRTK2CU7ZOQJQVQM5TQXB7JUYZPKRUERDWZIHJRFJJI4NQ, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.683131431 (85h37m54.405694461s) [noise=true , address=104.36.82.12:4183 , piece_id=3CBXKX64YWUKPNBLQIV664R52RKB5JJ3CTF3DORDXVODNZTXCJEQ, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.683362724 (85h37m54.405925764s) [noise=true , address=104.36.82.15:4282 , piece_id=7LJ4KOO2OFGNIKLFCQEXWRAQSYOLGY2HURCMZ24FZQ6F73H5JG6A, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.684539856 (85h37m54.407102896s) [noise=true , address=104.36.82.13:4201 , piece_id=FIWCSTUWTDNBCO5RXPFADOE4BKBBHAWMLHCHN6XPYDJVCKDFELNQ, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.685080721 (85h37m54.407643761s) [noise=true , address=104.36.82.55:4195 , piece_id=PLNEDUSO5JAUY4GMBNOXKDFJY2D5GYDU5HGBI5LEUPYXVMOVER7A, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.685375656 (85h37m54.407938686s) [noise=true , address=104.36.82.47:4520 , piece_id=55AXUVCPNXJWQ5ODSN6IMPTFON3EZCXIZKPDSM55XQUXI7PNCTDQ, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:44.685546442 (85h37m54.408109482s) [noise=true , address=77.90.31.11:4575 , piece_id=JHOFXWTB3AED35BU2PA5D7VLMVIFSTFEPDZJL45BPERIEXZPRMOA, seg_pos={0 0}, upload=82141]: Uploading piece... Jul 28 01:13:46.208544702 (85h37m55.931107742s) [upload=82141]: Next returned nil segment Jul 28 01:13:46.208561876 (85h37m55.931124916s) [upload=82141]: Waiting for error group managing segments... Jul 28 01:13:46.821725843 (85h37m56.544288883s) [noise=true , address=77.90.31.11:4575 , piece_id=JHOFXWTB3AED35BU2PA5D7VLMVIFSTFEPDZJL45BPERIEXZPRMOA, seg_pos={0 0}, upload=82141]: Done uploading piece. err: <nil> Jul 28 01:13:47.734268507 (85h37m57.456831557s) [noise=true , address=79.127.201.215:4360 , piece_id=FENQKHPRKPAJQUCEJU5LXULP4DMPKTNU6J4ZP46LURZ6MEQR5BTA, seg_pos={0 0}, upload=82141]: Done uploading piece. err: <nil> Jul 28 01:14:43.536778436 (85h38m53.259341476s) [noise=true , address=104.36.82.55:4195 , piece_id=PLNEDUSO5JAUY4GMBNOXKDFJY2D5GYDU5HGBI5LEUPYXVMOVER7A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: write tcp 188.166.225.170:48724->104.36.82.55:4195: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.536827911 (85h38m53.259390951s) [noise=true , address=104.36.82.47:4520 , piece_id=55AXUVCPNXJWQ5ODSN6IMPTFON3EZCXIZKPDSM55XQUXI7PNCTDQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: rpc: dial tcp 104.36.82.47:4520: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.536861275 (85h38m53.259424315s) [noise=true , address=104.36.82.10:4110 , piece_id=GWCKZ6SBCOGP6ORDW47NAB2IRMEW7IAJKY6GQ6DK2YGJYUR5QW2A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi, address:104.36.82.10:4110): protocol: write tcp 188.166.225.170:45934->104.36.82.10:4110: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537025509 (85h38m53.259588559s) [noise=true , address=104.36.82.49:4987 , piece_id=JKIKFUXRTK2CU7ZOQJQVQM5TQXB7JUYZPKRUERDWZIHJRFJJI4NQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:1oZxiaBQD1grLu8d1HyERweeuXNXc1aBgyAoe1B86gqm6ypvBC, address:104.36.82.49:4987): protocol: write tcp 188.166.225.170:59228->104.36.82.49:4987: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537133315 (85h38m53.259696355s) [noise=true , address=104.36.82.54:4166 , piece_id=SEH6RIIGR5JRPFYEKNVXC4PQAYY6PPZ53OOCHK2N3HSDLAA5LDBA, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 188.166.225.170:33120->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537190104 (85h38m53.259753134s) [noise=true , address=104.36.82.11:4141 , piece_id=QJMFURBAMGZVP7GSLRD2O5STKL6G3L7TWFALLCMPH2IOAUO32C7A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: rpc: dial tcp 104.36.82.11:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537260668 (85h38m53.259823708s) [noise=true , address=104.36.82.6:4885 , piece_id=KMGK5UBZGOCI5DZBGB657QDX24UQLTWR3P2RPLD7RVXZISLNAI5A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12rLd1AWyRtEVFmK4iJCebp5VAJEjnmhhgtUQcV1FB8vMvRnUhm, address:104.36.82.6:4885): protocol: write tcp 188.166.225.170:54480->104.36.82.6:4885: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537321495 (85h38m53.259884536s) [noise=true , address=104.36.82.16:4296 , piece_id=3MPXRJSVMDA24EEHGRWVDRBSDTA4RJ7SI35FVTLLWAU657QEFX7A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12cKhDbA4ptwux2LYNPNAJ7Qh8jECsGWd1KWaK6T6pD38bJWdt1, address:104.36.82.16:4296): protocol: write tcp 188.166.225.170:37180->104.36.82.16:4296: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537356312 (85h38m53.259919352s) [noise=true , address=104.36.82.20:4459 , piece_id=UOFLATHNK552SMXFAIF5KVMTBWFFGSP7BSYT5QK5PLSX74DUVG7Q, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: write tcp 188.166.225.170:52558->104.36.82.20:4459: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537386359 (85h38m53.259949389s) [noise=true , address=104.36.82.53:4140 , piece_id=FNO3QHP65GUKSQE6V4Z34ZT4O3QPZFC7IUEHJWPCRM7URP4UF34Q, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: rpc: dial tcp 104.36.82.53:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537414513 (85h38m53.259977543s) [noise=true , address=104.36.82.56:4241 , piece_id=MZ4L2GLXUQK7UNZU6N4K2TC4X6KMQLQRJQPMKSW2YP2J5LOY2IPA, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: rpc: dial tcp 104.36.82.56:4241: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537528812 (85h38m53.260091852s) [noise=true , address=104.36.82.14:4255 , piece_id=B3ZZOUK3LGO6Y4QFL57ILNXBKJLO45BFFJL5JWHRU6NLCRWNSWKQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 188.166.225.170:59488->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537560682 (85h38m53.260123722s) [noise=true , address=104.36.82.21:4484 , piece_id=WQNEJVZWO55EWNPKXTPPLQHE2XRC3N2N56JEO3RBBUFJ4LFJPRFQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12dQKfD74Hp3MiUYqMm2oxGfHiStWTXdg4c4MEq8gCZ1hcbkgaN, address:104.36.82.21:4484): protocol: rpc: dial tcp 104.36.82.21:4484: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537586973 (85h38m53.260150013s) [noise=true , address=104.36.82.12:4183 , piece_id=3CBXKX64YWUKPNBLQIV664R52RKB5JJ3CTF3DORDXVODNZTXCJEQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: write tcp 188.166.225.170:36568->104.36.82.12:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537600469 (85h38m53.260163509s) [noise=true , address=104.36.82.13:4201 , piece_id=FIWCSTUWTDNBCO5RXPFADOE4BKBBHAWMLHCHN6XPYDJVCKDFELNQ, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: rpc: dial tcp 104.36.82.13:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537658791 (85h38m53.260221831s) [noise=true , address=104.36.82.15:4282 , piece_id=7LJ4KOO2OFGNIKLFCQEXWRAQSYOLGY2HURCMZ24FZQ6F73H5JG6A, seg_pos={0 0}, upload=82141]: Done uploading piece. err: ecclient: upload failed (node:1jgMZEqSKy7o7LRHqAk3AU7qRhbMEjywnowLoe8n1suVkDoViB, address:104.36.82.15:4282): protocol: write tcp 188.166.225.170:59008->104.36.82.15:4282: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:14:43.537681464 (85h38m53.260244494s) [seg_pos={0 0}, upload=82141]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:14:43.779483383 (85h38m53.502046423s) [upload=82141]: Done uploading segments... Jul 28 01:14:43.779508130 (85h38m53.50207116s) [upload=82141]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.244908917s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X.......X.....X...............................X.....XXX...] duration: 5.836273915s FAIL (2025-07-28 01:13:51 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.836299736s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X...........X...................X..................X........] duration: 5.892017278s FAIL (2025-07-28 01:13:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.892041435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232426438s FAIL (2025-07-28 01:13:34 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ece_id=U5OXZBAGAHHQ5UFLNIPJYYWEM7T476LLFRFJBANFNTIWFB2JMBUA, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.913062898 (85h36m44.635625938s) [noise=true , address=104.36.82.13:4207 , piece_id=PGNRF2NDCX2UJ6QLIVOIJDG5QL3GTHUZ25RS6B75YXBUNMQHXDLA, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.913271407 (85h36m44.635834447s) [noise=true , address=104.36.82.10:4103 , piece_id=F422LAXFPYCCRY22BKPVQGOXWECU24SIW2LT3YRT3WOE44WYPUCQ, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.913473022 (85h36m44.636036062s) [noise=true , address=104.36.82.7:4927 , piece_id=GU6OFI7JPNEY2DOGGVQXK5HYVQKVXASFFGUBKCSHOSWVRBVDRRKA, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.913675730 (85h36m44.63623877s) [noise=true , address=104.36.82.56:4239 , piece_id=3AMHIXADDNIGBWGOQH3E7LSSGZOGA2WZ3FDT77RSWCU65TNG4A4A, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.913876224 (85h36m44.636439264s) [noise=true , address=104.36.82.11:4136 , piece_id=SS6GYCAWLWIRHDMFFK43IGYYLKK5AEVVOOSLJKKX5ASLX3IMJT5A, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.914055206 (85h36m44.636618246s) [noise=true , address=104.36.82.44:4894 , piece_id=G6LW366TWWTKMZKOQBDRJFNFCHGQC37ADDBFGVJ6YJQMR6XG7Y2Q, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.914253004 (85h36m44.636816044s) [noise=true , address=104.36.82.5:4843 , piece_id=AZIW72E2WRLHINB5AZ4HX3FF3ZILWORRHFJUSRR67P6PNMQKWDPQ, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:34.914451034 (85h36m44.637014074s) [noise=true , address=104.36.82.2:4758 , piece_id=XMOGKNYUNMCC5AVRKIPCYTLQO4H73GCJPRL3XBNUHNU355E6A4YQ, seg_pos={0 0}, upload=82126]: Uploading piece... Jul 28 01:12:36.444994309 (85h36m46.167557349s) [upload=82126]: Next returned nil segment Jul 28 01:12:36.445010480 (85h36m46.16757351s) [upload=82126]: Waiting for error group managing segments... Jul 28 01:12:38.621281396 (85h36m48.343844486s) [noise=true , address=79.127.201.215:4349 , piece_id=YEKX6JD3GQTNWUEFVZEFBDVVYU7UHZW56T4PS764Z5R32IA7MHJA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: <nil> Jul 28 01:12:46.121950701 (85h36m55.844513742s) [noise=true , address=64.49.13.137:4502 , piece_id=MDHHNJJESOHO5FGA2LKWRBV4POEK2FOWUSL4HJJLGBEFNMJB4C4Q, seg_pos={0 0}, upload=82126]: Done uploading piece. err: <nil> Jul 28 01:13:33.808303087 (85h37m43.530866127s) [noise=true , address=104.36.82.52:4106 , piece_id=BWKNLEZAH64OIE322HB474HKTNZKADXNUUODP2FNVI3R7PNKGNZQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: rpc: dial tcp 104.36.82.52:4106: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808351349 (85h37m43.530914389s) [noise=true , address=104.36.82.55:4220 , piece_id=KGJ4TFHVA7ERHLK2NP27MS2YWYNWHYHJ77FRSST3LXWOGXIUUPXQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:15jMrmLPcRL8VeYUPzYNPhc2N7qZt4QgTsrSHoo4Zrt4cGrHHK, address:104.36.82.55:4220): protocol: rpc: dial tcp 104.36.82.55:4220: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808385725 (85h37m43.530948755s) [noise=true , address=104.36.82.56:4239 , piece_id=3AMHIXADDNIGBWGOQH3E7LSSGZOGA2WZ3FDT77RSWCU65TNG4A4A, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808446071 (85h37m43.531009111s) [noise=true , address=104.36.82.21:4506 , piece_id=4JS3HYDJGEA62HPOB2PYCZGG4Z55G7I4LQGXIK6UBMVDN3Q4CJOQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:12YUTchJGJE2EGXqiGZEcLjQmFQCazjV8Ahi8ogJhuqTE4H6n7T, address:104.36.82.21:4506): protocol: rpc: dial tcp 104.36.82.21:4506: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808479895 (85h37m43.531042935s) [noise=true , address=104.36.82.5:4843 , piece_id=AZIW72E2WRLHINB5AZ4HX3FF3ZILWORRHFJUSRR67P6PNMQKWDPQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:12HvL2JDP4GCsu3tziVEKRMh14nEzyRMRuWNSmVvs27LzHTbNpS, address:104.36.82.5:4843): protocol: write tcp 188.166.225.170:44126->104.36.82.5:4843: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808515172 (85h37m43.531078212s) [noise=true , address=104.36.82.16:4328 , piece_id=CU5RXE5TIL2CAAJYKB2KJP56NXDO7PSMSWF7LOOJC3HBJA2UQZMA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1S8PSdkLyaiURWkPGhcABSh3481hoswzukT1NKvpTkSp75vKDJ, address:104.36.82.16:4328): protocol: rpc: dial tcp 104.36.82.16:4328: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808527346 (85h37m43.531090386s) [noise=true , address=104.36.82.11:4136 , piece_id=SS6GYCAWLWIRHDMFFK43IGYYLKK5AEVVOOSLJKKX5ASLX3IMJT5A, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:127n8CMbZQ7haiv6tGHrVg9V8Yf12sYkcbL9mEbSpQ4WkT9isnp, address:104.36.82.11:4136): protocol: write tcp 188.166.225.170:39968->104.36.82.11:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808547475 (85h37m43.531110515s) [noise=true , address=104.36.82.13:4207 , piece_id=PGNRF2NDCX2UJ6QLIVOIJDG5QL3GTHUZ25RS6B75YXBUNMQHXDLA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808562723 (85h37m43.531125754s) [noise=true , address=104.36.82.15:4270 , piece_id=AZTPMLXJ2JOCRVF3OXAYV2EVYOKYYEN37DGLQBGIKUF2OULU65GA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1nkuzy3vxeLdHhVs73Nceje7qXDiGJ1zEhpGgGKSk3cKb7vJbB, address:104.36.82.15:4270): protocol: rpc: dial tcp 104.36.82.15:4270: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808578414 (85h37m43.531141444s) [noise=true , address=104.36.82.7:4927 , piece_id=GU6OFI7JPNEY2DOGGVQXK5HYVQKVXASFFGUBKCSHOSWVRBVDRRKA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1AD5Ju87nY6LL5aTbFFsfNJRt8gkyGExsSrJLf6AP2q1nd9HBr, address:104.36.82.7:4927): protocol: rpc: dial tcp 104.36.82.7:4927: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808607670 (85h37m43.53117071s) [noise=true , address=104.36.82.41:4768 , piece_id=KIU5G5AXDQ77PHSJDAK33WAJSHQQ2IXKX6ZWZRMR6C2ZP7QLPX2A, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4, address:104.36.82.41:4768): protocol: rpc: dial tcp 104.36.82.41:4768: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808627588 (85h37m43.531190628s) [noise=true , address=104.36.82.10:4103 , piece_id=F422LAXFPYCCRY22BKPVQGOXWECU24SIW2LT3YRT3WOE44WYPUCQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:125eQFBenzeVGCxBzzDUQDNydyAntAWGWcq38CXqjmVt3qG5Qbn, address:104.36.82.10:4103): protocol: rpc: dial tcp 104.36.82.10:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808648968 (85h37m43.531212008s) [noise=true , address=104.36.82.54:4172 , piece_id=U5OXZBAGAHHQ5UFLNIPJYYWEM7T476LLFRFJBANFNTIWFB2JMBUA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd, address:104.36.82.54:4172): protocol: rpc: dial tcp 104.36.82.54:4172: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808657646 (85h37m43.531220686s) [noise=true , address=104.36.82.44:4894 , piece_id=G6LW366TWWTKMZKOQBDRJFNFCHGQC37ADDBFGVJ6YJQMR6XG7Y2Q, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: write tcp 188.166.225.170:52888->104.36.82.44:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808663086 (85h37m43.531226126s) [noise=true , address=104.36.82.14:4246 , piece_id=G5GFUWYQCLCS6QG62GYL6EAVFTRLMMQIIZXEI7QNKJGKG3RZUTGA, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:12dqLB6LVQJaSdEfjKmu5nXqRST733hn5rTxCcMQS8YJVv8Hwdc, address:104.36.82.14:4246): protocol: rpc: dial tcp 104.36.82.14:4246: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808672053 (85h37m43.531235093s) [noise=true , address=104.36.82.2:4758 , piece_id=XMOGKNYUNMCC5AVRKIPCYTLQO4H73GCJPRL3XBNUHNU355E6A4YQ, seg_pos={0 0}, upload=82126]: Done uploading piece. err: ecclient: upload failed (node:1joBbXdmusUXDfe1ofLp5oXy1nVkkRAng8QGL2hAtKjDz71u5g, address:104.36.82.2:4758): protocol: write tcp 188.166.225.170:51688->104.36.82.2:4758: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:13:33.808689977 (85h37m43.531253017s) [seg_pos={0 0}, upload=82126]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:13:34.039419739 (85h37m43.761982769s) [upload=82126]: Done uploading segments... Jul 28 01:13:34.039443435 (85h37m43.762006475s) [upload=82126]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232454341s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 1.81526598s FAIL (2025-07-28 01:13:07 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.815305836s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.X...........X...................X..................X.......] duration: 5.724660547s FAIL (2025-07-28 01:13:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.72468755s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...X.......X.....X...............................X.....XXX..] duration: 5.828457329s FAIL (2025-07-28 01:12:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.828489299s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 1.851210112s FAIL (2025-07-28 01:12:28 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.851243546s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236117546s FAIL (2025-07-28 01:12:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 82111]: Uploading piece... Jul 28 01:11:23.892790692 (85h35m33.615353732s) [noise=true , address=104.36.82.10:4107 , piece_id=G5PTR4LN63GUXVGIHAR7XSADECXWIIDYRWFWAKLMKVIH5QGPGG7A, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.893019850 (85h35m33.61558288s) [noise=true , address=104.36.82.55:4210 , piece_id=HZI5BXVVQZ6VBBDOWSW57OAKPUA2TFJ57EVIKWUVLUSLXPWIY2OA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.893207910 (85h35m33.61577095s) [noise=true , address=104.36.82.12:4154 , piece_id=2FLOIB3LUL5GCYLAOIYKZIFJ5GTUDIVQXHY4SJZ53ZVV4H2XKHJA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.893395720 (85h35m33.61595876s) [noise=true , address=104.36.82.11:4134 , piece_id=BDGTGNW2PARZIWZ6DUCDJ4WIJEOC4IQ3XA6Y7LSU4LBXBHJB4TWA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.893584702 (85h35m33.616147742s) [noise=true , address=104.36.82.42:4818 , piece_id=4CIHT7WAVREMAGASP66TVM7MFRBK4PTEQ63LXNL63OCQTWOGZWLQ, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.893790655 (85h35m33.616353695s) [noise=true , address=104.36.82.20:4450 , piece_id=ALWCLBUJS3IQKTQRLYF3QJSLPE6WB73EZN4CEOWIWH4TOYWKZ3BA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.894097613 (85h35m33.616660653s) [noise=true , address=104.36.82.56:4232 , piece_id=RPVBPRACIOUSYXDXK7JGFWFQTIU3ZP57RDZBSG5LDSXAWVZO4ZWA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.894302735 (85h35m33.616865775s) [noise=true , address=104.36.82.54:4180 , piece_id=4YN7LFRGOSN3ZUDRRIZ2DN7HN5AU7EMEGOLAKDL7AYSUNQNX3PIA, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:23.894493350 (85h35m33.61705639s) [noise=true , address=104.36.82.2:4739 , piece_id=SNH67PW43VMCAMFWG2A3PZGENQ5UPAX6ALJERCKUUV3VHCXQSXMQ, seg_pos={0 0}, upload=82111]: Uploading piece... Jul 28 01:11:25.018108166 (85h35m34.740671216s) [upload=82111]: Next returned nil segment Jul 28 01:11:25.018128444 (85h35m34.740691484s) [upload=82111]: Waiting for error group managing segments... Jul 28 01:11:25.397138379 (85h35m35.119701409s) [noise=true , address=79.127.205.246:4278 , piece_id=HXYHFOYVD64VTQP5EQD5JVOFWSHCGGSCYHQZM6EDN7CXD2TN222Q, seg_pos={0 0}, upload=82111]: Done uploading piece. err: <nil> Jul 28 01:11:29.671219304 (85h35m39.393782344s) [noise=true , address=79.127.201.215:4357 , piece_id=QGOQV456ASQM3IGENTWEULBHK527R7SLAJH5LTOJWNOJY5JLUQWA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: <nil> Jul 28 01:12:22.750004402 (85h36m32.472567442s) [noise=true , address=104.36.82.13:4207 , piece_id=ELBRE3TFHJBVXG5IQOPZY5EZ4NU4EN2EHG3Y73SSWR74JMBGY36A, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750058996 (85h36m32.472622036s) [noise=true , address=104.36.82.12:4154 , piece_id=2FLOIB3LUL5GCYLAOIYKZIFJ5GTUDIVQXHY4SJZ53ZVV4H2XKHJA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750097970 (85h36m32.47266101s) [noise=true , address=104.36.82.42:4818 , piece_id=4CIHT7WAVREMAGASP66TVM7MFRBK4PTEQ63LXNL63OCQTWOGZWLQ, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: rpc: dial tcp 104.36.82.42:4818: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750133328 (85h36m32.472696368s) [noise=true , address=104.36.82.56:4232 , piece_id=RPVBPRACIOUSYXDXK7JGFWFQTIU3ZP57RDZBSG5LDSXAWVZO4ZWA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1knYU6KSZJCmdpXTRHXZmCsRJYfNGGQCLPEPkWkBgUdyvtSgJe, address:104.36.82.56:4232): protocol: rpc: dial tcp 104.36.82.56:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750166692 (85h36m32.472729722s) [noise=true , address=104.36.82.2:4739 , piece_id=SNH67PW43VMCAMFWG2A3PZGENQ5UPAX6ALJERCKUUV3VHCXQSXMQ, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1eEbhCXRGArodr2pffqXQxRR8w6u4c6J6auDbsCwe1NH5Wwycd, address:104.36.82.2:4739): protocol: rpc: dial tcp 104.36.82.2:4739: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750197391 (85h36m32.472760431s) [noise=true , address=104.36.82.8:4003 , piece_id=V2MB2GQAFOD7ESWEYOP2D6PG6XRIJULGO7BFHWXT5FID3LINS7NQ, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1qTLdkqYYRGs1fpm8ZP17cGxPUYJCrspBwffuGbd6JbwcsRf65, address:104.36.82.8:4003): protocol: rpc: dial tcp 104.36.82.8:4003: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750228120 (85h36m32.47279116s) [noise=true , address=104.36.82.11:4134 , piece_id=BDGTGNW2PARZIWZ6DUCDJ4WIJEOC4IQ3XA6Y7LSU4LBXBHJB4TWA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12R3Z5AgkBmWXMv5XgRyfcyqkG2gmyJs9ogT24kcLy6nuB9bzUG, address:104.36.82.11:4134): protocol: rpc: dial tcp 104.36.82.11:4134: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750316198 (85h36m32.472879238s) [noise=true , address=104.36.82.55:4210 , piece_id=HZI5BXVVQZ6VBBDOWSW57OAKPUA2TFJ57EVIKWUVLUSLXPWIY2OA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750373658 (85h36m32.472936698s) [noise=true , address=104.36.82.53:4152 , piece_id=TO22UZ4SOJBFGQSQ5MIXEY7LHL7GYLFL42DSZQNMTREVAWA3DNFQ, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: rpc: dial tcp 104.36.82.53:4152: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750415408 (85h36m32.472978448s) [noise=true , address=104.36.82.20:4450 , piece_id=ALWCLBUJS3IQKTQRLYF3QJSLPE6WB73EZN4CEOWIWH4TOYWKZ3BA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12UeL1LySKQySa8wjvG4FBs69cE1iWkW4Diz83nu1PxBCKxyvki, address:104.36.82.20:4450): protocol: rpc: dial tcp 104.36.82.20:4450: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750451848 (85h36m32.473014888s) [noise=true , address=104.36.82.52:4102 , piece_id=MPYJIPJNM6OF4OBTFKX7HLLRGW5R5FJ3BOUB3G2LKQNYGJG2B2JA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: rpc: dial tcp 104.36.82.52:4102: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750484209 (85h36m32.473047249s) [noise=true , address=104.36.82.15:4269 , piece_id=QOS2INSZUMB5MSMUSONVWKEFVS24TKL4SDSGVA5MU5KCESYFLYEA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12V8v5nzyyouSqndATi4fRJUC6HgFbhguBjnmwpki48zdRBTcgC, address:104.36.82.15:4269): protocol: rpc: dial tcp 104.36.82.15:4269: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750524768 (85h36m32.473087808s) [noise=true , address=104.36.82.54:4180 , piece_id=4YN7LFRGOSN3ZUDRRIZ2DN7HN5AU7EMEGOLAKDL7AYSUNQNX3PIA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750533314 (85h36m32.473096354s) [noise=true , address=104.36.82.10:4107 , piece_id=G5PTR4LN63GUXVGIHAR7XSADECXWIIDYRWFWAKLMKVIH5QGPGG7A, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:104.36.82.10:4107): protocol: rpc: dial tcp 104.36.82.10:4107: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750560125 (85h36m32.473123165s) [noise=true , address=104.36.82.41:4786 , piece_id=ZJX6GMX6LIUCBRQYGRDZZHWC5RGKUP4NO5LRQDW4NEDVKW2HDQOA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q, address:104.36.82.41:4786): protocol: write tcp 188.166.225.170:50384->104.36.82.41:4786: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750576957 (85h36m32.473139998s) [noise=true , address=104.36.82.44:4894 , piece_id=7UD7IRVRLO7F23XWPXIPHFXUZW4Z4S5M4ILIDYIASOVUAG5QJNCA, seg_pos={0 0}, upload=82111]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:104.36.82.44:4894): protocol: rpc: dial tcp 104.36.82.44:4894: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:12:22.750599360 (85h36m32.4731624s) [seg_pos={0 0}, upload=82111]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:12:22.984443210 (85h36m32.70700625s) [upload=82111]: Done uploading segments... Jul 28 01:12:22.984475924 (85h36m32.707038964s) [upload=82111]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.236131292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.X...........X...................X..................X......] duration: 5.800086705s FAIL (2025-07-28 01:12:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.800132794s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 1.882171651s FAIL (2025-07-28 01:11:44 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.882200065s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [....X.......X.....X...............................X.....XXX.] duration: 5.660277113s FAIL (2025-07-28 01:11:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.660312992s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.237213511s FAIL (2025-07-28 01:11:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 03758683s) [noise=true , address=104.36.82.9:4013 , piece_id=Z6QHMCZLVG4Z3NABLR2NKWZKCANM6BUTMQ6VQKA4MPZCFPNNUAWQ, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.381414191 (85h34m33.103977231s) [noise=true , address=104.36.82.42:4825 , piece_id=IRMGK5ZNQXQF5UFS7YELJ2ZPOVBZI2YVNYSYH6Y2B672LYFO33AA, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.381626437 (85h34m33.104189487s) [noise=true , address=104.36.82.11:4148 , piece_id=4YBAZRTYW3L7QX7DYBNN4E3O5R5J6YHLDLIRANAIFKRQB4V4HWGA, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.381689658 (85h34m33.104252698s) [noise=true , address=104.36.82.12:4165 , piece_id=XQSUYUEXV6QVJRTV5H3ZZVUWVRWQSZ7F2TAL5I5MERLBIGRIFTQA, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.382029608 (85h34m33.104592648s) [noise=true , address=64.49.13.138:4482 , piece_id=JRMNNK2ZPXHZ6DW7LBX5JTLVUUT2VTEGNLCCJWAN3ZBP2ERN3Y5Q, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.382311728 (85h34m33.104874769s) [noise=true , address=104.36.82.10:4103 , piece_id=BMPD3KQJMUPR6BMJGG27B5GYVEMM7SNTN47PJ4RVE2QC2AKNQWKA, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.382421809 (85h34m33.104984849s) [noise=true , address=104.36.82.41:4778 , piece_id=J4QMKRV3FQFV7XF3GVU2KRE5VKN6FOHLYU6MA2CIGE36CO5V6UYQ, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:23.382657049 (85h34m33.105220159s) [noise=true , address=104.36.82.53:4138 , piece_id=4CJHOAHFJTBZ3KIZH7N4NJDC2XR3M6IZ3J2YNFQRR6HROFI2SI4A, seg_pos={0 0}, upload=82095]: Uploading piece... Jul 28 01:10:24.559616197 (85h34m34.282179237s) [upload=82095]: Next returned nil segment Jul 28 01:10:24.559631066 (85h34m34.282194096s) [upload=82095]: Waiting for error group managing segments... Jul 28 01:10:24.860821399 (85h34m34.583384439s) [noise=true , address=64.49.13.138:4482 , piece_id=JRMNNK2ZPXHZ6DW7LBX5JTLVUUT2VTEGNLCCJWAN3ZBP2ERN3Y5Q, seg_pos={0 0}, upload=82095]: Done uploading piece. err: <nil> Jul 28 01:11:22.287344416 (85h35m32.009907456s) [noise=true , address=104.36.82.21:4473 , piece_id=BWEUKF3AKUIWM5NLWLQAJ6D26Z4A2PGBBTYULL3L53HE4O5ITNZQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12DDe4jT3YeUH8qMQgapcvzMnFs8UFB8fTmubho4ZQf5tYVoxuk, address:104.36.82.21:4473): protocol: rpc: dial tcp 104.36.82.21:4473: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287396385 (85h35m32.009959425s) [noise=true , address=104.36.82.44:4873 , piece_id=JIWZAGAS36QR2QKLLS7VSSLOFLIRDAR27ZR6INB6BO6CSBR7EEGQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12b5GdYtmYz9D1QW2z78W3Z2fWvMJGtrvrMJ1njbcqbtCDUdwHo, address:104.36.82.44:4873): protocol: rpc: dial tcp 104.36.82.44:4873: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287432765 (85h35m32.009995805s) [noise=true , address=104.36.82.9:4013 , piece_id=Z6QHMCZLVG4Z3NABLR2NKWZKCANM6BUTMQ6VQKA4MPZCFPNNUAWQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:127jXtnz4wJDoxR5x8ebMqRt94RvbxMZm4YzzZrabbVohZpF8eW, address:104.36.82.9:4013): protocol: rpc: dial tcp 104.36.82.9:4013: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287462262 (85h35m32.010025302s) [noise=true , address=104.36.82.12:4165 , piece_id=XQSUYUEXV6QVJRTV5H3ZZVUWVRWQSZ7F2TAL5I5MERLBIGRIFTQA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:13Ksy7YCGV5u3T5Z6WwKtMjFq6Rs4bayzR7mQFyXbSFuJjaH4F, address:104.36.82.12:4165): protocol: rpc: dial tcp 104.36.82.12:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287490546 (85h35m32.010053586s) [noise=true , address=104.36.82.51:4071 , piece_id=EB6PGOVTFJSAHN7RDISQAMTDE4ONPUSU3PUCPRJOMVOTJ6PQZVMA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:1VqztUNDN3hhJAMvCJtPreW7BdtAWSofQxAtmDKfDiX5sNEfWY, address:104.36.82.51:4071): protocol: rpc: dial tcp 104.36.82.51:4071: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287520904 (85h35m32.010083944s) [noise=true , address=104.36.82.45:4929 , piece_id=QUZ3DNQSITKBACNT3PMXTVVWCATZLKVKPQNGU5F5HSARRW263YTQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm, address:104.36.82.45:4929): protocol: rpc: dial tcp 104.36.82.45:4929: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287551793 (85h35m32.010114833s) [noise=true , address=104.36.82.54:4180 , piece_id=THHPKLRSR2PJDB2CFSCFOG4XPXF26SARWDCUND42QT7B4L35IKAA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3, address:104.36.82.54:4180): protocol: rpc: dial tcp 104.36.82.54:4180: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287581360 (85h35m32.0101444s) [noise=true , address=104.36.82.41:4778 , piece_id=J4QMKRV3FQFV7XF3GVU2KRE5VKN6FOHLYU6MA2CIGE36CO5V6UYQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: rpc: dial tcp 104.36.82.41:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287614202 (85h35m32.010177242s) [noise=true , address=104.36.82.53:4138 , piece_id=4CJHOAHFJTBZ3KIZH7N4NJDC2XR3M6IZ3J2YNFQRR6HROFI2SI4A, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: write tcp 188.166.225.170:49356->104.36.82.53:4138: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287655101 (85h35m32.010218141s) [noise=true , address=104.36.82.11:4148 , piece_id=4YBAZRTYW3L7QX7DYBNN4E3O5R5J6YHLDLIRANAIFKRQB4V4HWGA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12Rkh9uZUA5jXBVZvvt1jE7GHiEszKvR9PweKEry2AkdwoCiANs, address:104.36.82.11:4148): protocol: write tcp 188.166.225.170:39084->104.36.82.11:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287687793 (85h35m32.010250833s) [noise=true , address=104.36.82.47:4539 , piece_id=2KKHLPKFVXCVPZACTNBGJCCAAF5GFVZLFHHD425SPP3IFEOZJIBA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:126g75XgAyFYcgnaibseqoGiz7cK6z3n6gPMcu9TrdjuZUYEHnZ, address:104.36.82.47:4539): protocol: rpc: dial tcp 104.36.82.47:4539: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287715637 (85h35m32.010278677s) [noise=true , address=104.36.82.3:4769 , piece_id=UH4TGJBDR3FGD5ELIKK6J2KPKQQZMSZC6DHCFCKQ4HQVDI3OTPJA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:1CNSBmPmCSQ6gHKNcWrFAy3VU7Rf2ukMYQpkdurgZLPmqotWQ6, address:104.36.82.3:4769): protocol: rpc: dial tcp 104.36.82.3:4769: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287760051 (85h35m32.010323091s) [noise=true , address=104.36.82.10:4103 , piece_id=BMPD3KQJMUPR6BMJGG27B5GYVEMM7SNTN47PJ4RVE2QC2AKNQWKA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:125eQFBenzeVGCxBzzDUQDNydyAntAWGWcq38CXqjmVt3qG5Qbn, address:104.36.82.10:4103): protocol: rpc: dial tcp 104.36.82.10:4103: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287796141 (85h35m32.010359191s) [noise=true , address=104.36.82.56:4249 , piece_id=USOYBFOS3F4W5ZGKVSYIEMBI75GPT4RNNJDNZQP2B4WMZWL7LAQQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: rpc: dial tcp 104.36.82.56:4249: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287810488 (85h35m32.010373528s) [noise=true , address=104.36.82.20:4463 , piece_id=DRGI7MITIUELW5OY5U4ZBTQN3I6HTJ5UAD4UPVEC6DP6BB3E6YOA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12vZnQQMq6WQKtsDgDvoE6Qwgp3cPbDSHMZzhCnBuJk1Rx8dT4c, address:104.36.82.20:4463): protocol: rpc: dial tcp 104.36.82.20:4463: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287828943 (85h35m32.010391983s) [noise=true , address=104.36.82.50:4028 , piece_id=7QXNMEYKKJAF7PTAQEERH5D7TK6QVST5XHX2SUZFBVRCD7TOS7NQ, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12B8N15ijqmYJVmjCT74JTLcMEYSZ4xdAWxVqRRF6w8hgYJkmHF, address:104.36.82.50:4028): protocol: rpc: dial tcp 104.36.82.50:4028: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287837830 (85h35m32.01040087s) [noise=true , address=104.36.82.42:4825 , piece_id=IRMGK5ZNQXQF5UFS7YELJ2ZPOVBZI2YVNYSYH6Y2B672LYFO33AA, seg_pos={0 0}, upload=82095]: Done uploading piece. err: ecclient: upload failed (node:12AYyhgZh7oAaGf4EXLSp6s1csFxCKfLuh6CJAPyzdC6pLQ51h, address:104.36.82.42:4825): protocol: rpc: dial tcp 104.36.82.42:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:11:22.287855284 (85h35m32.010418324s) [seg_pos={0 0}, upload=82095]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:11:22.522195003 (85h35m32.244758043s) [upload=82095]: Done uploading segments... Jul 28 01:11:22.522222025 (85h35m32.244785065s) [upload=82095]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.237235843s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X....X.......X.....X...............................X.....XXX] duration: 30.244410304s FAIL (2025-07-28 01:11:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.244433168s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.X...........X...................X..................X.....] duration: 6.033901153s FAIL (2025-07-28 01:11:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.033927041s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 1.86517941s FAIL (2025-07-28 01:11:08 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.865206291s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.X...........X...................X..................X....] duration: 5.816891531s FAIL (2025-07-28 01:10:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.816919975s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.233618033s FAIL (2025-07-28 01:10:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== 33m32.132016126s) [noise=true , address=104.36.82.56:4256 , piece_id=YGQIG55DUSYSV7ECINGIYUL7TBAG7SST3GK5LAHEDHYQTESDNTQA, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.409656105 (85h33m32.132219145s) [noise=true , address=104.36.82.12:4165 , piece_id=CJE5LBRBDX2QEHPW2TOYBJX3ESFAL5LORPKAO5RQMWREOJFTRSLQ, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.409917114 (85h33m32.132480154s) [noise=true , address=104.36.82.52:4105 , piece_id=SUOUKGBDZPSJCXTPSDN4AL6WIXFL4GAKTNRNXQ7RI4NHF2DV7UDQ, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.410306769 (85h33m32.132869809s) [noise=true , address=104.36.82.5:4871 , piece_id=77ZT4DHVLV37CHBAGBN4U5U4BCBG3RXJ2PXNMZB3M3OAGS4SMIXQ, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.410630068 (85h33m32.133193108s) [noise=true , address=104.36.82.44:4880 , piece_id=ICTJ33U2KN6SWSNO32XWM2GAFIV7SEXKH2S3XSLCCOT5XS3U4OZA, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.410965419 (85h33m32.133528459s) [noise=true , address=104.36.82.53:4121 , piece_id=ICTHL62Q46RBNYVAFYFA3DBJB5S53RRE3ZFZ52QCZAUZAL64JRGA, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.411278549 (85h33m32.133841589s) [noise=true , address=104.36.82.11:4143 , piece_id=JNNWRKWWYNDC5HKQFUSHARVNEG7KZZ5QTQ36DH7MMHOXV73BTRCA, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:22.411563834 (85h33m32.134126874s) [noise=true , address=104.36.82.46:4512 , piece_id=JVUCLNXUE35DKO5ML7R4WADD6V22LSJ5OORR5UNVMYER7WIFHH5A, seg_pos={0 0}, upload=82082]: Uploading piece... Jul 28 01:09:23.937926030 (85h33m33.660489071s) [upload=82082]: Next returned nil segment Jul 28 01:09:23.937947351 (85h33m33.660510392s) [upload=82082]: Waiting for error group managing segments... Jul 28 01:09:25.950733174 (85h33m35.673296214s) [noise=true , address=79.127.201.215:4345 , piece_id=UGWEVXZWBNYWCOFXWLWPJBMME4JRBKP656RHL7EHE63YVPUTMSAQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: <nil> Jul 28 01:10:21.316403498 (85h34m31.038966538s) [noise=true , address=104.36.82.49:4996 , piece_id=4QV27XHTSMYEMDMX2U6S5Z6LN3IMFSJMV5EFYS5W4THRUTIIDFPQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1qomJ2ceswE6SKfoPAxWpiCkcW88EpfFbPnJXs7ETb5CZvSpqS, address:104.36.82.49:4996): protocol: rpc: dial tcp 104.36.82.49:4996: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316543545 (85h34m31.039106585s) [noise=true , address=104.36.82.7:4920 , piece_id=MHGFFA3B777QDGJ34MBEXLXS45MMI5E7FRZV6FXMP22PX26655LQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1vBccAgciQwbLaMMe9QKfYM3zxYTHqG3pfYQ7nh8eETr8JUskb, address:104.36.82.7:4920): protocol: rpc: dial tcp 104.36.82.7:4920: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316581297 (85h34m31.039144337s) [noise=true , address=104.36.82.2:4737 , piece_id=IGMMT5C2KR3Y5KEU324FIPF5PNCOJTX2LK3JGELBJU66UBWNMA2A, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:12jSyKX7N1ngVoXmADe6qBrVVoc4CVjHYfhsHqENf4cUPmNTnAh, address:104.36.82.2:4737): protocol: rpc: dial tcp 104.36.82.2:4737: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316617997 (85h34m31.039181037s) [noise=true , address=104.36.82.56:4256 , piece_id=YGQIG55DUSYSV7ECINGIYUL7TBAG7SST3GK5LAHEDHYQTESDNTQA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:11N1BR6rgiYye7CnuM5xDJogUXoUp4FgXrXM41GdhfrRCbbgT8, address:104.36.82.56:4256): protocol: rpc: dial tcp 104.36.82.56:4256: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316653907 (85h34m31.039216947s) [noise=true , address=104.36.82.5:4871 , piece_id=77ZT4DHVLV37CHBAGBN4U5U4BCBG3RXJ2PXNMZB3M3OAGS4SMIXQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1NEVGFZtdgMqREnYBPuBbYqwGRxExenRwncwCBLDFhuwdFyWdb, address:104.36.82.5:4871): protocol: rpc: dial tcp 104.36.82.5:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316689124 (85h34m31.039252164s) [noise=true , address=104.36.82.53:4121 , piece_id=ICTHL62Q46RBNYVAFYFA3DBJB5S53RRE3ZFZ52QCZAUZAL64JRGA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: rpc: dial tcp 104.36.82.53:4121: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316721495 (85h34m31.039284535s) [noise=true , address=104.36.82.46:4512 , piece_id=JVUCLNXUE35DKO5ML7R4WADD6V22LSJ5OORR5UNVMYER7WIFHH5A, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1rUdrYUJgCF8aa3MdNe82dT765nqW46by7mJ2phMSpfHAhzgU2, address:104.36.82.46:4512): protocol: rpc: dial tcp 104.36.82.46:4512: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316768987 (85h34m31.039332027s) [noise=true , address=104.36.82.14:4235 , piece_id=SJRDUEVWW5OXTQORAOOXQNWDJZ3P3YLZ56SB6RG2BAZVS5J3DGGA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1nn8EX7mq6mXic78Jura9BcAHfgw8rFwzfqf5HtwJtrYt6PUDV, address:104.36.82.14:4235): protocol: rpc: dial tcp 104.36.82.14:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316827558 (85h34m31.039390599s) [noise=true , address=104.36.82.52:4105 , piece_id=SUOUKGBDZPSJCXTPSDN4AL6WIXFL4GAKTNRNXQ7RI4NHF2DV7UDQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1TVtMQNVV2kERp2aZwmS8E9JUiua1RCyFpZgFACRqH1Nfscw4C, address:104.36.82.52:4105): protocol: rpc: dial tcp 104.36.82.52:4105: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.316873617 (85h34m31.039436657s) [noise=true , address=104.36.82.44:4880 , piece_id=ICTJ33U2KN6SWSNO32XWM2GAFIV7SEXKH2S3XSLCCOT5XS3U4OZA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: rpc: dial tcp 104.36.82.44:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317016520 (85h34m31.03957956s) [noise=true , address=104.36.82.11:4143 , piece_id=JNNWRKWWYNDC5HKQFUSHARVNEG7KZZ5QTQ36DH7MMHOXV73BTRCA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317052799 (85h34m31.039615839s) [noise=true , address=104.36.82.16:4311 , piece_id=T6L2IPW2NL342FDDNMAO5DHRGUX5LJ5HTKV6MLOLWYPBZ4G5PFOQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:12qPZjuPpHwR2HwuBRK3F1QjN5zdU7StDPtN2PZ1ZeCP1ebqYmH, address:104.36.82.16:4311): protocol: rpc: dial tcp 104.36.82.16:4311: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317086063 (85h34m31.039649103s) [noise=true , address=104.36.82.55:4225 , piece_id=3BXZMJM22GUK7I2ID6AFEP2EA5K6FOP5UU4XRLSKDJWKGLX76F5Q, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:12HoBC3rCo4dTHYKsXZJNfSETHzkihdGfATJe46LdWM3DXVRdtR, address:104.36.82.55:4225): protocol: rpc: dial tcp 104.36.82.55:4225: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317121441 (85h34m31.039684481s) [noise=true , address=104.36.82.50:4042 , piece_id=QM64HWD5L3CBDGBTYGUV5QAY2V47YVOBJ3KNTPPK4KOYIHTC52FA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1LbRrZsqp1TR354jZQLstDmWAUBVBMyid8AuAbTfQj7N89Mnte, address:104.36.82.50:4042): protocol: rpc: dial tcp 104.36.82.50:4042: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317158401 (85h34m31.039721442s) [noise=true , address=104.36.82.13:4199 , piece_id=WRJ2KTVBPNREV55436XJCBJRA7NXR5AB47CPJRQIOPDPR472ZRMA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:1DagQMPpe9F6w5xewXXCfzVBPcmgg6EKCauFsNCWHoHdP56bs7, address:104.36.82.13:4199): protocol: write tcp 188.166.225.170:40646->104.36.82.13:4199: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317216833 (85h34m31.039779873s) [noise=true , address=104.36.82.12:4165 , piece_id=CJE5LBRBDX2QEHPW2TOYBJX3ESFAL5LORPKAO5RQMWREOJFTRSLQ, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:13Ksy7YCGV5u3T5Z6WwKtMjFq6Rs4bayzR7mQFyXbSFuJjaH4F, address:104.36.82.12:4165): protocol: rpc: dial tcp 104.36.82.12:4165: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317251870 (85h34m31.03981491s) [noise=true , address=104.36.82.15:4278 , piece_id=NIJPGCVONOQUVJANS6UKIKY2BVFH4R5NGKU6PRHNTCS4IUBG3ZKA, seg_pos={0 0}, upload=82082]: Done uploading piece. err: ecclient: upload failed (node:12qpbyKnJ8qXmos7gdxd46F8QXx98LUwtKD6mryaXPBDD5WFSwu, address:104.36.82.15:4278): protocol: write tcp 188.166.225.170:39988->104.36.82.15:4278: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:10:21.317268522 (85h34m31.039831572s) [seg_pos={0 0}, upload=82082]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:10:21.547218501 (85h34m31.269781542s) [upload=82082]: Done uploading segments... Jul 28 01:10:21.547243950 (85h34m31.26980698s) [upload=82082]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.233629504s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X....X.......X.....X...............................X.....XX] duration: 30.235651376s FAIL (2025-07-28 01:10:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.235677688s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 1.84498385s FAIL (2025-07-28 01:10:05 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.845014208s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.X...........X...................X..................X...] duration: 5.830568869s FAIL (2025-07-28 01:09:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.830596353s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X....X.......X.....X...............................X.....X] duration: 30.236084361s FAIL (2025-07-28 01:09:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.236110391s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239758233s FAIL (2025-07-28 01:09:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== MLK6WJ4VGIP4RHA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.967946711 (85h32m31.690509751s) [noise=true , address=104.36.82.56:4239 , piece_id=V3YIAB6VQUF5K3STKFK3TCUDVUKQ3OXEZE6562ZLSMYWNVLLKI5Q, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.968013177 (85h32m31.690576217s) [noise=true , address=104.36.82.11:4122 , piece_id=2OOHU7B3W7IPT5FOQQHW4HRYN4AFFGORM5A65ELHKHDQPS6RSKIQ, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.968453340 (85h32m31.69101638s) [noise=true , address=104.36.82.15:4262 , piece_id=AUVGQRKQUMEUGZGEQDRJOVFAF6IHTIRI7NOEPXPB54GYTUKMKOXA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.968873985 (85h32m31.691437025s) [noise=true , address=104.36.82.14:4253 , piece_id=OY7X2EXWCFZRF7YWI3NSS7JGCNT7I4JG2IBNISFSPLYPTTQQGXPA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.969314027 (85h32m31.691877057s) [noise=true , address=104.36.82.45:4929 , piece_id=B65SNMGC2N2CNV2CPJQZJ54VLE7YESSK7TE3YTS67VAKY5XCMIGA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.969710946 (85h32m31.692273986s) [noise=true , address=104.36.82.12:4171 , piece_id=BN5QJQCHIVYB776QPD3GHWTO3PBC2FYWND2INCI2YXVOOZGYQITA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.970460800 (85h32m31.69302384s) [noise=true , address=104.36.82.10:4086 , piece_id=AGRVDHC6BQB6E3B7KBKEF3ALUGGF24B6DZHWCHVOTFUNC3XPX2EA, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:21.970781524 (85h32m31.693344564s) [noise=true , address=104.36.82.22:4510 , piece_id=SRHRMVP6L2JMK3P3EYIOXEHK4QK5XBFSDDXYP3XVVQKQTTQERKHQ, seg_pos={0 0}, upload=82069]: Uploading piece... Jul 28 01:08:23.142501398 (85h32m32.865064438s) [upload=82069]: Next returned nil segment Jul 28 01:08:23.142522157 (85h32m32.865085197s) [upload=82069]: Waiting for error group managing segments... Jul 28 01:08:24.735078803 (85h32m34.457641843s) [noise=true , address=79.127.205.233:4656 , piece_id=E7XBXZMQPTR7RAQYIDRZJQSA4LPCVHA7TK6UR6ZZJOLKZ6PBG7TQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: <nil> Jul 28 01:09:20.848768342 (85h33m30.571331382s) [noise=true , address=104.36.82.15:4262 , piece_id=AUVGQRKQUMEUGZGEQDRJOVFAF6IHTIRI7NOEPXPB54GYTUKMKOXA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:104.36.82.15:4262): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:09:20.848822315 (85h33m30.571385355s) [noise=true , address=104.36.82.53:4129 , piece_id=W4BOJJODXCSH36LLBM364BFZ3YKE6SCQ2OO3L5TB3WCH4INS2AAQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848838927 (85h33m30.571401977s) [noise=true , address=104.36.82.14:4253 , piece_id=OY7X2EXWCFZRF7YWI3NSS7JGCNT7I4JG2IBNISFSPLYPTTQQGXPA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12vJmwywMNTHh1ce5JGzbMiUBA3FijDecNEJ1TDU1Wyd4NWzA14, address:104.36.82.14:4253): protocol: rpc: dial tcp 104.36.82.14:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848879144 (85h33m30.571442184s) [noise=true , address=104.36.82.45:4929 , piece_id=B65SNMGC2N2CNV2CPJQZJ54VLE7YESSK7TE3YTS67VAKY5XCMIGA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12VxQN6CV1ogwKcs27ho2ayknnPi7BDEcA9ZFbgsUacuQNFAdmm, address:104.36.82.45:4929): protocol: rpc: dial tcp 104.36.82.45:4929: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848908910 (85h33m30.57147195s) [noise=true , address=104.36.82.52:4085 , piece_id=RO52AF5SPIIPZRC3OGHKCKKSUP4UX4K2JF3BJIXPGOVCDOL4YOJA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: rpc: dial tcp 104.36.82.52:4085: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848931123 (85h33m30.571494163s) [noise=true , address=104.36.82.44:4891 , piece_id=VEETEUZZAREG4H5HNQWK7RZYPF76U7XLC7MUYBN7BVHIUCEV25CQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848980468 (85h33m30.571543498s) [noise=true , address=104.36.82.16:4312 , piece_id=SUDSUMZEH52MUI727LA4NB5J2FVTLDWXEFYSCCXNPQWLIDTNU6BQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12AVp3qn5uwLPqYaZEwzcgoGMJhNga1BUh6bK5k2bxjWgZLQiiC, address:104.36.82.16:4312): protocol: rpc: dial tcp 104.36.82.16:4312: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.848991278 (85h33m30.571554318s) [noise=true , address=104.36.82.8:4000 , piece_id=QMIBYLVBBYK5U7FAREJC7H4JEI6YTDKLB3BN5MLK6WJ4VGIP4RHA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12rr6Tv4yJszNcSqx1j8K4VCXVFQkRNzqwVxh9cWRkEe3HYXtd2, address:104.36.82.8:4000): protocol: rpc: dial tcp 104.36.82.8:4000: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849039841 (85h33m30.571602891s) [noise=true , address=104.36.82.12:4171 , piece_id=BN5QJQCHIVYB776QPD3GHWTO3PBC2FYWND2INCI2YXVOOZGYQITA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1z2Fh5fPFTb9VZX6to5hdUYUGst63Ti1BVtdVtYk7vkrUqXMKM, address:104.36.82.12:4171): protocol: rpc: dial tcp 104.36.82.12:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849083205 (85h33m30.571646245s) [noise=true , address=104.36.82.11:4122 , piece_id=2OOHU7B3W7IPT5FOQQHW4HRYN4AFFGORM5A65ELHKHDQPS6RSKIQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1oCu75zR8YccTunHjctcLzTrr9wTQqtAPXHiWrsQs41o8gxgh6, address:104.36.82.11:4122): protocol: rpc: dial tcp 104.36.82.11:4122: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849093283 (85h33m30.571656324s) [noise=true , address=104.36.82.10:4086 , piece_id=AGRVDHC6BQB6E3B7KBKEF3ALUGGF24B6DZHWCHVOTFUNC3XPX2EA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1fMyQ9ojAvrZhG4sLwG3wzNJZihg8ZwWKBJmgWpK3Ckx5YXrg3, address:104.36.82.10:4086): protocol: rpc: dial tcp 104.36.82.10:4086: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849110377 (85h33m30.571673417s) [noise=true , address=104.36.82.22:4510 , piece_id=SRHRMVP6L2JMK3P3EYIOXEHK4QK5XBFSDDXYP3XVVQKQTTQERKHQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9, address:104.36.82.22:4510): protocol: rpc: dial tcp 104.36.82.22:4510: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849132699 (85h33m30.571695729s) [noise=true , address=104.36.82.3:4773 , piece_id=JRQJTCM4SI4UPMFBGD7EV24K3K6736V7TZCF7CMVWV4GCHX5AC3Q, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1RdN5fHVgcrtHFNS6AdzSgGavbtJntKXpEimi8Pvf2DPFPBPQV, address:104.36.82.3:4773): protocol: write tcp 188.166.225.170:40260->104.36.82.3:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849149962 (85h33m30.571713002s) [noise=true , address=104.36.82.6:4896 , piece_id=F7YAF4E3N3NZ2DSB2PC4KGFCDT73EZRSNJQPCGV4PTW5FK2W7GJQ, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1XD8SwxzT9F6zcfBtCL6dCtBm3hyPMXgo5ooJ8oM2UL1wdnT7f, address:104.36.82.6:4896): protocol: rpc: dial tcp 104.36.82.6:4896: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849154400 (85h33m30.57171744s) [noise=true , address=104.36.82.13:4196 , piece_id=LJLZUJIO55WPTSG4OE5O7YMGJ5AJ2L4K6NSLIHJL3P3RMJWDUB5Q, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:1aLZa6kv5VhDgqvUZp2HkzpkbA81KU13iGTTsiVfSeUDzLu8Lu, address:104.36.82.13:4196): protocol: rpc: dial tcp 104.36.82.13:4196: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849159280 (85h33m30.57172232s) [noise=true , address=104.36.82.56:4239 , piece_id=V3YIAB6VQUF5K3STKFK3TCUDVUKQ3OXEZE6562ZLSMYWNVLLKI5Q, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849168788 (85h33m30.571731828s) [noise=true , address=104.36.82.55:4215 , piece_id=CTP4ZQSTRUEGAOBZ3MGSKCZH7CURHPQTIUCWSQXPZ7W6PE6PV4YA, seg_pos={0 0}, upload=82069]: Done uploading piece. err: ecclient: upload failed (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR, address:104.36.82.55:4215): protocol: rpc: dial tcp 104.36.82.55:4215: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:09:20.849201430 (85h33m30.571764471s) [seg_pos={0 0}, upload=82069]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:09:21.086541660 (85h33m30.80910469s) [upload=82069]: Done uploading segments... Jul 28 01:09:21.086567299 (85h33m30.809130339s) [upload=82069]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.239770968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 1.892022759s FAIL (2025-07-28 01:09:10 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.89206013s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 1.883899529s FAIL (2025-07-28 01:08:31 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.88392121s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.X...........X...................X..................X..] duration: 5.884573893s FAIL (2025-07-28 01:08:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.884614502s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [..XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.241688402s FAIL (2025-07-28 01:08:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== A5WXFMYN2Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.506865408 (85h31m31.229428448s) [noise=true , address=104.36.82.55:4197 , piece_id=QGDRS6XKQZFT43LDPBXZ74AI3H47FHMNMDSULORKBBMVV5JRYM6Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.507058748 (85h31m31.229621788s) [noise=true , address=104.36.82.41:4786 , piece_id=Q4A7PAHXQPQSMPHXPTAPLE5EN3GRGGP4STNW2CNBNYTUEITE5POA, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.507147567 (85h31m31.229710607s) [noise=true , address=104.36.82.9:4029 , piece_id=4AXZPVZ3VVXJXZJWDRCXNEBA72LHG4VUMORFJ6ITKTMSV454P5VQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.509846261 (85h31m31.232409301s) [noise=true , address=104.36.82.35:4613 , piece_id=AAFXLFHNY7BCIXJDXR3CIAL5QWDUWHVCPWNLEYMIZA4KELHKEX3A, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.510247798 (85h31m31.232810828s) [noise=true , address=104.36.82.15:4271 , piece_id=LA4U3W42JQQSGTWZACUXJEFWOAS3TKAYRFGSJXIYEIHNY6ENFDKQ, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:21.512143535 (85h31m31.234706576s) [noise=true , address=104.36.82.7:4917 , piece_id=BIQ7H3DYWOAMVA3USH4H77H5L5MMGJ4TZCVLLM3FCJWRSSLW4S6Q, seg_pos={0 0}, upload=82052]: Uploading piece... Jul 28 01:07:23.115672703 (85h31m32.838235743s) [upload=82052]: Next returned nil segment Jul 28 01:07:23.115688904 (85h31m32.838251944s) [upload=82052]: Waiting for error group managing segments... Jul 28 01:07:23.592484560 (85h31m33.3150476s) [noise=true , address=45.91.116.9:4277 , piece_id=JOW7N4LFI2ZW2MB4TWWVVUTPQSNNJEMZRRJPOYYYWJ7FP3OFK5NQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: <nil> Jul 28 01:08:20.374838590 (85h32m30.09740163s) [noise=true , address=104.36.82.44:4875 , piece_id=FNSR2C52CPG7YOCDQDBS3XR2IPELPWCJWO2E4TCQD4X67LJU3ZPA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1, address:104.36.82.44:4875): protocol: rpc: dial tcp 104.36.82.44:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.374892704 (85h32m30.097455734s) [noise=true , address=104.36.82.4:4830 , piece_id=B7KS3EB2H6BHT5DDNDTSY3VXZ7USBV55LW42VC7U7NNP6VGBHZPQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1p91PtS64Ue8JkPmbh2f3gig8cTQ3AMdDJPaP1249h4XmXUJrp, address:104.36.82.4:4830): protocol: write tcp 188.166.225.170:60856->104.36.82.4:4830: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.374924655 (85h32m30.097487695s) [noise=true , address=104.36.82.16:4324 , piece_id=TPCOBZ3GMWDNDCZXHX66NUKZOCEQ2VLQJXAQG3NIWQ46JOPTZDLA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12tmAP2k9LHn7tGVu5Ky9pLpwQ4aXettco3TRZ1fbmEC5SESkga, address:104.36.82.16:4324): protocol: rpc: dial tcp 104.36.82.16:4324: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.374954672 (85h32m30.097517712s) [noise=true , address=104.36.82.55:4197 , piece_id=QGDRS6XKQZFT43LDPBXZ74AI3H47FHMNMDSULORKBBMVV5JRYM6Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12niHLkT5pbmYSbarHZGeeCRtwjZRXtvEtHMGAb7ja4a7Xuuvfi, address:104.36.82.55:4197): protocol: write tcp 188.166.225.170:43708->104.36.82.55:4197: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.374983367 (85h32m30.097546407s) [noise=true , address=104.36.82.9:4029 , piece_id=4AXZPVZ3VVXJXZJWDRCXNEBA72LHG4VUMORFJ6ITKTMSV454P5VQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12snsP1y5fbxY8XrDoXihzQrsBxJ5Y38C2jSGhHtV1ukN426eiP, address:104.36.82.9:4029): protocol: rpc: dial tcp 104.36.82.9:4029: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375012854 (85h32m30.097575894s) [noise=true , address=104.36.82.35:4613 , piece_id=AAFXLFHNY7BCIXJDXR3CIAL5QWDUWHVCPWNLEYMIZA4KELHKEX3A, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1xjm7XPPwM9nXBP3SS36HeLNxBVJdEpcQyD9BAp79xbGCze4vL, address:104.36.82.35:4613): protocol: write tcp 188.166.225.170:34332->104.36.82.35:4613: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375047470 (85h32m30.0976105s) [noise=true , address=104.36.82.13:4214 , piece_id=XHE3IV6EMDSNU2A6RMNJHLPDQYVRXQ77JASN4FIV4TR3PGGLI3NA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1nJ8uqwxf6x4hX52AjAdCbMKi1BKGGbMvrQbiazJdLpn6j69ay, address:104.36.82.13:4214): protocol: write tcp 188.166.225.170:58280->104.36.82.13:4214: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375079752 (85h32m30.097642792s) [noise=true , address=104.36.82.3:4779 , piece_id=OYELVXCKDF6PBRSQP7AMICRTMGCHEUCDH4M4ITTGNOIU7DZ6SXHQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12DHKWSFHcN6FYabUR9Q8vpe7duLcyZmEPcDmqvvcQHDGm4bpvu, address:104.36.82.3:4779): protocol: write tcp 188.166.225.170:40806->104.36.82.3:4779: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375270417 (85h32m30.097833457s) [noise=true , address=104.36.82.14:4232 , piece_id=KRZU2DIWHE4F6KOQ6OVV3C63JWNS2V24IW3JNDRBQYJQ6XBEE7UA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1jWewg226E4BrBxqcK38n4ZjHx3toQeNdbc5HeU2EoykztXwyX, address:104.36.82.14:4232): protocol: rpc: dial tcp 104.36.82.14:4232: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375336893 (85h32m30.097899933s) [noise=true , address=104.36.82.46:4481 , piece_id=5CI2SQJWKALC5DZWXYTKXIR5O7KHB2EZ2UWLSLW5NOCMKQIPOD7Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:47504->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375367993 (85h32m30.097931043s) [noise=true , address=104.36.82.22:4525 , piece_id=KZGEC3VE6SNGGLTCE4NEVRDOXT4OASI6R3SE6BBDYMTTJMDAZFDQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12Yx3qg5NAW8iAw56mkwALjkkhyFF5feKgTZxcTG31RxBsubh5A, address:104.36.82.22:4525): protocol: write tcp 188.166.225.170:48084->104.36.82.22:4525: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375404553 (85h32m30.097967593s) [noise=true , address=104.36.82.56:4263 , piece_id=YEKH5Z7DRANHUQIPMIGBJEO3UJ5CTFQ4PK5Q3KZYIXA5WXFMYN2Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: write tcp 188.166.225.170:51550->104.36.82.56:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375435843 (85h32m30.097998883s) [noise=true , address=104.36.82.41:4786 , piece_id=Q4A7PAHXQPQSMPHXPTAPLE5EN3GRGGP4STNW2CNBNYTUEITE5POA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:125TWU7R4pkmrUxsNLhgTD475aLbjUJ7psA6drJtE7oDqZXYJ4Q, address:104.36.82.41:4786): protocol: rpc: dial tcp 104.36.82.41:4786: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375464999 (85h32m30.098028039s) [noise=true , address=104.36.82.7:4917 , piece_id=BIQ7H3DYWOAMVA3USH4H77H5L5MMGJ4TZCVLLM3FCJWRSSLW4S6Q, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1VaqXNoBYjwbbE9xhAoLzDuxZMdBPSm2aMw7bYm4qZqiEGp5CV, address:104.36.82.7:4917): protocol: rpc: dial tcp 104.36.82.7:4917: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375490568 (85h32m30.098053608s) [noise=true , address=104.36.82.54:4183 , piece_id=WAKXJMDZGQKZZGUQFO2DATTKNEFUFWAUIVVWUIII2SJ6BJMWL4OQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:52488->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375504213 (85h32m30.098067263s) [noise=true , address=104.36.82.15:4271 , piece_id=LA4U3W42JQQSGTWZACUXJEFWOAS3TKAYRFGSJXIYEIHNY6ENFDKQ, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:1T9uQjynbTYcaixhvHWCKzatRWpFxMonHFi5idiMQtvivupKSu, address:104.36.82.15:4271): protocol: rpc: dial tcp 104.36.82.15:4271: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375510356 (85h32m30.098073396s) [noise=true , address=104.36.82.53:4152 , piece_id=LKX5S67FE5WLX5NQVNGE3T37VRVIZMN7N2BKS37TU6A3ZUQRGRAA, seg_pos={0 0}, upload=82052]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:42504->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:08:20.375524022 (85h32m30.098087062s) [seg_pos={0 0}, upload=82052]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 01:08:20.614289920 (85h32m30.33685296s) [upload=82052]: Done uploading segments... Jul 28 01:08:20.614309156 (85h32m30.336872186s) [upload=82052]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.241700415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X....X.......X.....X...............................X.....] duration: 5.859303025s FAIL (2025-07-28 01:08:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.859355125s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 1.979813586s FAIL (2025-07-28 01:08:04 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.979850116s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.X...........X...................X..................X.] duration: 5.830322348s FAIL (2025-07-28 01:07:53 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.830347877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X....X.......X.....X...............................X....] duration: 5.765248032s FAIL (2025-07-28 01:07:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.765302686s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X.....X.X...........X...................X..................X] duration: 30.233592927s FAIL (2025-07-28 01:07:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ece_id=BWQXEJ6OLUJINPDYLHHUYPI3HF7ERY5BUW3I3RDCBMWLH5YYYSPA, seg_pos={0 0}, upload=82041]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: context canceled; context canceled Jul 28 01:06:37.286095979 (85h30m47.00865924s) [seg_pos={0 0}, upload=82041]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 01:06:37.286153449 (85h30m47.008716489s) [upload=82041]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:06:37.286158539 (85h30m47.008721579s) [upload=82041]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:06:37.571898070 (85h30m47.29446112s) [upload=82041]: Done uploading segments... Jul 28 01:06:37.571954458 (85h30m47.294517548s) [upload=82041]: Upload finished. err: <nil> Jul 28 01:07:18.325449141 (85h31m28.048012181s) [upload=82050]: Starting upload Jul 28 01:07:18.325525046 (85h31m28.048088086s) [upload=82050]: Done starting upload Jul 28 01:07:18.325604138 (85h31m28.048167178s) [upload=82050]: Uploading segments... Jul 28 01:07:18.325716072 (85h31m28.048279112s) [upload=82050]: Got next segment. Inline: false Jul 28 01:07:18.325748985 (85h31m28.048312025s) [upload=82050]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:07:18.325754164 (85h31m28.048317204s) [upload=82050]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:07:18.566715451 (85h31m28.289278491s) [seg_pos={0 0}, upload=82050]: Begin upload segment... Jul 28 01:07:18.566759406 (85h31m28.289322436s) [seg_pos={0 0}, upload=82050]: Done begin upload segment. Jul 28 01:07:18.566781338 (85h31m28.289344368s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.566870408 (85h31m28.289433448s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567166264 (85h31m28.289729304s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567283709 (85h31m28.289846739s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567430209 (85h31m28.289993249s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567551361 (85h31m28.290114401s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567685177 (85h31m28.290248207s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567807421 (85h31m28.290370461s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.567949422 (85h31m28.290512462s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568055656 (85h31m28.290618696s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568209150 (85h31m28.29077218s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568316686 (85h31m28.290879716s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568534673 (85h31m28.291097713s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568566183 (85h31m28.291129223s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568767778 (85h31m28.291330818s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:18.568787867 (85h31m28.291350907s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Uploading piece... Jul 28 01:07:20.263761425 (85h31m29.986324465s) [upload=82050]: Next returned nil segment Jul 28 01:07:20.263778748 (85h31m29.986341789s) [upload=82050]: Waiting for error group managing segments... Jul 28 01:07:20.586370989 (85h31m30.308934029s) [noise=true , address=45.91.116.6:4973 , piece_id=JK6IT7VX4NK5ESUA3Y7SJWFNV4CCI5GUO3ZXL5NYV6NRYS6KBLZA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.604357032 (85h31m30.326920072s) [noise=true , address=45.91.116.12:4225 , piece_id=ZZW3T4KTULBAEIESRJOFZNXGHG3AHA3YUXNGZKAPEY5XFK5AFGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.646286845 (85h31m30.368849885s) [noise=true , address=45.91.116.14:4190 , piece_id=NHZPLJ2YKVTD4MPJ4CF7DW4FLPDCT645AWAGRNAPF2ICPOCFL7EQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.660311178 (85h31m30.382874228s) [noise=true , address=77.90.31.12:4590 , piece_id=MDWH6BVJ3FKMMPXIZII7GU45CULABIFR6OWEJG3OVEG7W7CLHWXA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.662441384 (85h31m30.385004424s) [noise=true , address=45.91.116.12:4227 , piece_id=ASZLZDMSDRNID63SSTX37QIJMOBIDQJ2NH4AQJFQXXODIF7GMINQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.683513491 (85h31m30.406076531s) [noise=true , address=45.91.116.10:4081 , piece_id=R4347Q24OTM5DJMKO4BBDUTOB2SO333JSTBSHB2IFQWXA5SU4ENA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.688354693 (85h31m30.410917733s) [noise=true , address=45.91.116.7:4010 , piece_id=3MOXA5BN2IGUZCQLAYJ5V4IPFCTS3KRC35YML7FC5GXT7OS7QGDA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.697381383 (85h31m30.419944423s) [noise=true , address=45.91.116.9:4263 , piece_id=BEOKGJGFF3GTPBUAXQWR3AEVBC4LAQTNWEQL65SSPHU5NTWESJJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.704099167 (85h31m30.426662207s) [noise=true , address=45.91.116.5:4298 , piece_id=IZZ6YF77ZX7SS3F6WREOD6TGJ7PYB5CJXIE5LPIDCOJEFXZ65ISQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.715487876 (85h31m30.438050917s) [noise=true , address=77.90.31.16:4734 , piece_id=4FL5PR4PQBUWPMH4HYNVUF4BXJBLFSEERGZFNRSXVP4R4FEC7W4Q, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.746084446 (85h31m30.468647486s) [noise=true , address=77.90.31.19:4843 , piece_id=OCSJ3F53PVGDGJ5T7TDOKP6M62SHEUO3H6QFBROPYZVVOTR36JJA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:20.788608206 (85h31m30.511171246s) [noise=true , address=45.91.116.6:4975 , piece_id=HHZ5GA63AFR7YAEQ6Q7YJ2SW5OAGXK5HA5FVXISHDHTCXSPYVU2A, seg_pos={0 0}, upload=82050]: Done uploading piece. err: <nil> Jul 28 01:07:47.676947870 (85h31m57.3995109s) [noise=true , address=104.36.82.2:4732 , piece_id=XZ4PHPXB2QW7U7P2AFZLO5UDPJMX6U3XPFD6HP6CJZGD7SI7FZPQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: rpc: dial tcp 104.36.82.2:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.676994299 (85h31m57.399557369s) [noise=true , address=104.36.82.6:4875 , piece_id=WOOGDXGJONBWD6VEZBEOMGRUQXOFB2TGMQ6O4JTNT6GWP54JZ5WA, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 188.166.225.170:33566->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677005340 (85h31m57.39956837s) [noise=true , address=104.36.82.16:4293 , piece_id=ONXJD6AMBSGJABQPWF36K4OJLTTZMDCDGU7YBMHMWWD3KKDF6LRQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 188.166.225.170:60550->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677014647 (85h31m57.399577687s) [noise=true , address=104.36.82.7:4911 , piece_id=UUODVGKTZ5VGVPO2E753O6JEYGPIDQBVZVFD7DEKFUOJLLY54KSQ, seg_pos={0 0}, upload=82050]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:60150->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:47.677031981 (85h31m57.399595011s) [seg_pos={0 0}, upload=82050]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:07:47.908620335 (85h31m57.631183375s) [upload=82050]: Done uploading segments... Jul 28 01:07:47.908645694 (85h31m57.631208734s) [upload=82050]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.233616623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.244110195s FAIL (2025-07-28 01:07:20 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== ADM3O2ZQBRGAJWVA, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.059435464 (85h30m30.781998504s) [noise=true , address=104.36.82.56:4239 , piece_id=JRES5HWWHVQDLSW2ASE5LYNC752TU3S2IUP4YEQDHDD7FZ6ZJWEQ, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.059682316 (85h30m30.782245356s) [noise=true , address=104.36.82.21:4491 , piece_id=XLADSRNPQUAHQNT6LTOHWQNFVCLZWMLGRYI7TVXCTVP6EMSS2JOQ, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.059697915 (85h30m30.782260955s) [noise=true , address=104.36.82.49:4000 , piece_id=OJR4BWRZUP6ANCO6JVXOEOBR3WGRKCAUDVSVBUHPEWEMDEU7R7KA, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.059899752 (85h30m30.782462792s) [noise=true , address=104.36.82.53:4133 , piece_id=DVV6YDNXF2YDGSE64UA7I24ICTJG6HFS6NMVWZZBZMT3A2ZUBRPA, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.059950439 (85h30m30.782513479s) [noise=true , address=104.36.82.46:4503 , piece_id=H6E2MUHBPVDXOBBME5MDROOKYZRRE3ZBRXHZXFZP7NYG4BQ5ANCA, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.060135262 (85h30m30.782698302s) [noise=true , address=104.36.82.51:4080 , piece_id=3VI76D5XK7DWTOP2FD26ZJRBSSCCPVIL4X2KRV4NY3VRTIY5HQ2Q, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.060202692 (85h30m30.782765732s) [noise=true , address=104.36.82.5:4861 , piece_id=5FBSFDBLGRFBLXR53P5Q3WRF7CJM2CWM3Q5QFJPZXMF6CLIVY3JA, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:06:21.060350905 (85h30m30.782913945s) [noise=true , address=104.36.82.12:4179 , piece_id=L4YWVI2QYKHNFL5R6TSCI4VM2PWX4HZJ6XCP5HYQXCD7ER66TNOQ, seg_pos={0 0}, upload=82037]: Uploading piece... Jul 28 01:07:19.898004788 (85h31m29.620567848s) [upload=82037]: Next segment err: context deadline exceeded Jul 28 01:07:19.899071639 (85h31m29.621634679s) [noise=true , address=104.36.82.53:4133 , piece_id=DVV6YDNXF2YDGSE64UA7I24ICTJG6HFS6NMVWZZBZMT3A2ZUBRPA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899137476 (85h31m29.621700516s) [noise=true , address=104.36.82.47:4551 , piece_id=MGOWDPBNHUDISEJG43A2S3UZ5TLXAJAPW2EL6A7RNKSHSMC2XQOA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1UhJvSMtNqJDak4TM7eUgz1N7yCn4ChmRRTe7nXdffTNsS1pch, address:104.36.82.47:4551): protocol: rpc: dial tcp 104.36.82.47:4551: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899169166 (85h31m29.621732206s) [noise=true , address=104.36.82.52:4115 , piece_id=DWNYATJLOKRT2OIWQJRPPTKXZC6KOKU7XNHJ6XNDUB2QPFCR5U5A, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: rpc: dial tcp 104.36.82.52:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899199905 (85h31m29.621762945s) [noise=true , address=104.36.82.52:4096 , piece_id=7YDVKMS7IZFBUO4JSH22QLBIJTEINWEAFAODBASBHTYI4ULD7SQQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899283004 (85h31m29.621846044s) [noise=true , address=104.36.82.55:4224 , piece_id=VNTMQI57W3V2PYV7YFLA2UX3BKFYHVO4YE5OUHDVB56XQBIAWRQA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899346185 (85h31m29.621909225s) [noise=true , address=104.36.82.21:4491 , piece_id=XLADSRNPQUAHQNT6LTOHWQNFVCLZWMLGRYI7TVXCTVP6EMSS2JOQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:12Qxcfg1HCcyaydtJb4i3CtKaweXzyRZxkght7xRePgMzXX1QF6, address:104.36.82.21:4491): protocol: rpc: dial tcp 104.36.82.21:4491: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899394086 (85h31m29.621957117s) [noise=true , address=104.36.82.51:4080 , piece_id=3VI76D5XK7DWTOP2FD26ZJRBSSCCPVIL4X2KRV4NY3VRTIY5HQ2Q, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: rpc: dial tcp 104.36.82.51:4080: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899434964 (85h31m29.621998034s) [noise=true , address=104.36.82.12:4179 , piece_id=L4YWVI2QYKHNFL5R6TSCI4VM2PWX4HZJ6XCP5HYQXCD7ER66TNOQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:129pP9f3rFfTvP6MhG894N7WHuJRZP8aWhzJf1zBwZ76wbzVbZa, address:104.36.82.12:4179): protocol: rpc: dial tcp 104.36.82.12:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899625218 (85h31m29.622188248s) [noise=true , address=104.36.82.13:4204 , piece_id=BOVZPFVMASZT35RY22NXDDMWZTNRBRBW5Q2PSXOWLEABW627C2LQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899748775 (85h31m29.622311825s) [noise=true , address=104.36.82.11:4135 , piece_id=XSG4OOXPTXMQUREAX3ZJJ3CB5CND5YVYUYFFSJ5YBAOKWBKEYONQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1uEBWRaMH4mLApUcgHyHQmGxxbyXnYUScg5Z2on9LHESYkqj3S, address:104.36.82.11:4135): protocol: rpc: dial tcp 104.36.82.11:4135: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899805123 (85h31m29.622368153s) [noise=true , address=104.36.82.56:4239 , piece_id=JRES5HWWHVQDLSW2ASE5LYNC752TU3S2IUP4YEQDHDD7FZ6ZJWEQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899854037 (85h31m29.622417077s) [noise=true , address=104.36.82.22:4542 , piece_id=4D5UXWEE6GUNC3XEAZOWPDCNGB5AF5NZWJ6J7CQAN3DLSRBUIZWA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1V1MxSKuV5ZxsXXJBppZ5qmFwJPovFp8wD1biE1sF7mWzQ9qpL, address:104.36.82.22:4542): protocol: rpc: dial tcp 104.36.82.22:4542: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.899923569 (85h31m29.622486609s) [noise=true , address=104.36.82.14:4254 , piece_id=3AYVKAKZL4KRG7LZXA77SZUTAW4XDZEYVWDJADM3O2ZQBRGAJWVA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1bH5wgoULxFcdaLErG9JUnkbSfkLxv3zYu7VYhpAkRfQV2YJ91, address:104.36.82.14:4254): protocol: rpc: dial tcp 104.36.82.14:4254: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900034481 (85h31m29.622597532s) [noise=true , address=104.36.82.49:4000 , piece_id=OJR4BWRZUP6ANCO6JVXOEOBR3WGRKCAUDVSVBUHPEWEMDEU7R7KA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:12chhfe1weiPC1jjv8vbScP7uFVNXFr6s3UcNRkH8gsSAtGD16D, address:104.36.82.49:4000): protocol: write tcp 188.166.225.170:37876->104.36.82.49:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900124443 (85h31m29.622687483s) [noise=true , address=104.36.82.5:4861 , piece_id=5FBSFDBLGRFBLXR53P5Q3WRF7CJM2CWM3Q5QFJPZXMF6CLIVY3JA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: rpc: dial tcp 104.36.82.5:4861: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900164791 (85h31m29.622727821s) [noise=true , address=104.36.82.42:4827 , piece_id=LMBQLF6HMEOS6VSTCWZH4SHTKELMMYZW7AAKNAK2GBGVKLQHX2KQ, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1pqAbf2u3J689bYTQUaaxWdF4R241i7qYHraSWK14nV68pSUfC, address:104.36.82.42:4827): protocol: rpc: dial tcp 104.36.82.42:4827: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900197183 (85h31m29.622760223s) [noise=true , address=104.36.82.10:4094 , piece_id=JM4JTX5SJHSMR3OFA5IS3JI7QN5254HXHCWOHXLVUYO4WISDAXQA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:124zBCfZBTW69BSaGRzUyjWUBjSJhaLMZXQYnqMPGSKL4EX6HVV, address:104.36.82.10:4094): protocol: rpc: dial tcp 104.36.82.10:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900234324 (85h31m29.622797364s) [noise=true , address=104.36.82.46:4503 , piece_id=H6E2MUHBPVDXOBBME5MDROOKYZRRE3ZBRXHZXFZP7NYG4BQ5ANCA, seg_pos={0 0}, upload=82037]: Done uploading piece. err: ecclient: upload failed (node:1ctmSF1aMQasuXByFW8DppDantREmmWL2WUjmKGW5ZULVePyvm, address:104.36.82.46:4503): protocol: write tcp 188.166.225.170:40396->104.36.82.46:4503: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:07:19.900247319 (85h31m29.622810349s) [seg_pos={0 0}, upload=82037]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.244125213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 1.893779259s FAIL (2025-07-28 01:07:16 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.893812052s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.....X.X...........X...................X..................] duration: 6.065680205s FAIL (2025-07-28 01:06:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.065707185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [X....X....X.......X.....X...............................X...] duration: 5.842775334s FAIL (2025-07-28 01:06:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.842814078s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 57 out of 60 is failed (threshold=3) [X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.410169609s FAIL (2025-07-28 01:06:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== piece_id=XTYE6KOXMZIISWJPD2VAJHAO2NZXXPLIMPJ7VC3W2J3IF5MG54AA, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.393781841 (85h29m30.116344901s) [noise=true , address=104.36.82.4:4822 , piece_id=ZSM4OUXBNJG7VWYCRTPAQJN2TSUCL3TQEPTPRVR7KM55OP5QHBUA, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.394042609 (85h29m30.116605659s) [noise=true , address=104.36.82.56:4247 , piece_id=2TN3LYY27BC7MMGEO2VA6B4UX5YERQ6PPWLZOGL6NAZZ52TTX2UQ, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.394192967 (85h29m30.116756007s) [noise=true , address=104.36.82.54:4185 , piece_id=LCGI5IV7E3KPA4RYML2M46KQWZ7BTIGP3KEDCBREIWUVDEUSAZ3Q, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.394334207 (85h29m30.116897257s) [noise=true , address=104.36.82.21:4490 , piece_id=4IKDVUUFXREBWNXBRLZK3IYZEP4HTRJF465OA3YYCEZMTBTAUWFA, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.394536484 (85h29m30.117099534s) [noise=true , address=104.36.82.53:4122 , piece_id=HYBZJZC7G33VRO62N4UHG4OAWNQPJQ5MYCZ7F3PXVPH72JQDZD6A, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.394765623 (85h29m30.117328673s) [noise=true , address=79.127.201.215:4357 , piece_id=XKMMTRPBJ2HQ6FPWZIR2KVRDD5VKL2UWKWFF2YQI2T55O2J3WGRQ, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.395106896 (85h29m30.117669936s) [noise=true , address=79.127.205.245:4249 , piece_id=UCD4OBY5XWCOTTZKCPKG7NXOLXHWGKOKCQUPMFMP4OQORH3XXV2Q, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:20.399504219 (85h29m30.122067269s) [noise=true , address=104.36.82.11:4141 , piece_id=3HZSDWNLKWYNKNVU45F2MLPMIX3PIQXWSAHQRVUG4SZNMFHJ4LFQ, seg_pos={0 0}, upload=82020]: Uploading piece... Jul 28 01:05:21.518348176 (85h29m31.240911216s) [upload=82020]: Next returned nil segment Jul 28 01:05:21.518365770 (85h29m31.24092881s) [upload=82020]: Waiting for error group managing segments... Jul 28 01:05:22.001713552 (85h29m31.724276602s) [noise=true , address=79.127.205.245:4249 , piece_id=UCD4OBY5XWCOTTZKCPKG7NXOLXHWGKOKCQUPMFMP4OQORH3XXV2Q, seg_pos={0 0}, upload=82020]: Done uploading piece. err: <nil> Jul 28 01:05:30.186235474 (85h29m39.908798514s) [noise=true , address=79.127.201.215:4357 , piece_id=XKMMTRPBJ2HQ6FPWZIR2KVRDD5VKL2UWKWFF2YQI2T55O2J3WGRQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: <nil> Jul 28 01:06:19.253997817 (85h30m28.976560857s) [noise=true , address=104.36.82.7:4923 , piece_id=CYLIEP52T5KJW3WSFU22HSRHMF6CFZQBNL73DZQQOYH3FIBA4BHQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12Ee9ZfH81oy4SAUxozjQNWJ7ALmWDXQZYPLFwuAiJ8V6E7jT6r, address:104.36.82.7:4923): protocol: rpc: dial tcp 104.36.82.7:4923: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254066820 (85h30m28.97662986s) [noise=true , address=104.36.82.14:4227 , piece_id=DKDBE2GE6DHJMC7NYDWQYBJ7HO5BGOYSVIIYHTL5KG6UXYTQ5KDA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12nwrhLqAKoDy7KkoPsgCAy44m7T1JcEqbgatNYXThZ7sGc84vA, address:104.36.82.14:4227): protocol: write tcp 188.166.225.170:46040->104.36.82.14:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254127636 (85h30m28.976690676s) [noise=true , address=104.36.82.11:4141 , piece_id=3HZSDWNLKWYNKNVU45F2MLPMIX3PIQXWSAHQRVUG4SZNMFHJ4LFQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:52062->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254270469 (85h30m28.976833499s) [noise=true , address=104.36.82.4:4822 , piece_id=ZSM4OUXBNJG7VWYCRTPAQJN2TSUCL3TQEPTPRVR7KM55OP5QHBUA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:104.36.82.4:4822): protocol: rpc: dial tcp 104.36.82.4:4822: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254338490 (85h30m28.97690152s) [noise=true , address=104.36.82.56:4247 , piece_id=2TN3LYY27BC7MMGEO2VA6B4UX5YERQ6PPWLZOGL6NAZZ52TTX2UQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: rpc: dial tcp 104.36.82.56:4247: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254384677 (85h30m28.976947717s) [noise=true , address=104.36.82.22:4541 , piece_id=AYWM4MUQIYFT5TYPTEK547CK4YB3OR5UYT5GM7SILNH3WZRMPMAA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1zupYUhZUbuwMv5gwprk8M3kPYQ6qJiFK8ZsgGg3b1sqH6UwGr, address:104.36.82.22:4541): protocol: rpc: dial tcp 104.36.82.22:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254416840 (85h30m28.97697988s) [noise=true , address=104.36.82.13:4212 , piece_id=VFE4BQC26KZSNGR5WZOYMWKD5LC3QXBTNMJLXH2Y2LHG74FJH3ZQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254448390 (85h30m28.97701143s) [noise=true , address=104.36.82.54:4185 , piece_id=LCGI5IV7E3KPA4RYML2M46KQWZ7BTIGP3KEDCBREIWUVDEUSAZ3Q, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: rpc: dial tcp 104.36.82.54:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254522892 (85h30m28.977085932s) [noise=true , address=104.36.82.21:4490 , piece_id=4IKDVUUFXREBWNXBRLZK3IYZEP4HTRJF465OA3YYCEZMTBTAUWFA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12DPUW1Ej5FLg9yTbhqdWBX8YAryWG82BnVegsCiTRX71pUA24n, address:104.36.82.21:4490): protocol: rpc: dial tcp 104.36.82.21:4490: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254619527 (85h30m28.977182567s) [noise=true , address=104.36.82.20:4441 , piece_id=UXSFQPJLIDMF5VBBKPXHF22OLTSSIWX34JRHMDPALQLYMNQSBYQQ, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12LjGecRNJE1kq67iMtxZFhWeaUZpUogsGxbz4VXwnHdQtTnbBr, address:104.36.82.20:4441): protocol: write tcp 188.166.225.170:55076->104.36.82.20:4441: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254663111 (85h30m28.977226151s) [noise=true , address=104.36.82.42:4825 , piece_id=7GXDWFQRIO3EGZDGTKUNCI72HTUXIDA3GTOLBYNY6HAKHVNYG7CA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:12AYyhgZh7oAaGf4EXLSp6s1csFxCKfLuh6CJAPyzdC6pLQ51h, address:104.36.82.42:4825): protocol: rpc: dial tcp 104.36.82.42:4825: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254673069 (85h30m28.97723611s) [noise=true , address=104.36.82.17:4331 , piece_id=UIRSUKBKULKY6HG2BFDIZVMEFDMSI3AYNWIFXW26MW2AP4AZGG3A, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1JCBWFkdqHkXb7CR2vdPeHWPuMRRoZT2HWqTa1zDwQ9gwfu8uU, address:104.36.82.17:4331): protocol: rpc: dial tcp 104.36.82.17:4331: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254702866 (85h30m28.977265906s) [noise=true , address=104.36.82.6:4904 , piece_id=Z4UJ7ISZUYL2GGPVT2AOCSFPM4KPLAX7WU5V4IE4RO7RIZWGUV3A, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1y8S5sEV2o6s823Jb5f11EmDHGbQaFkWZ18k9TCii3TR5vRt9y, address:104.36.82.6:4904): protocol: rpc: dial tcp 104.36.82.6:4904: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254724668 (85h30m28.977287708s) [noise=true , address=104.36.82.55:4207 , piece_id=XTYE6KOXMZIISWJPD2VAJHAO2NZXXPLIMPJ7VC3W2J3IF5MG54AA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254738955 (85h30m28.977301995s) [noise=true , address=104.36.82.15:4263 , piece_id=PNVQZLZ2JWGWKRNPGSNM66UYMC3WKKUQBYAIIFPZ6Q5VNKEXGFQA, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: rpc: dial tcp 104.36.82.15:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254767510 (85h30m28.97733055s) [noise=true , address=104.36.82.53:4122 , piece_id=HYBZJZC7G33VRO62N4UHG4OAWNQPJQ5MYCZ7F3PXVPH72JQDZD6A, seg_pos={0 0}, upload=82020]: Done uploading piece. err: ecclient: upload failed (node:124d52DhtbtwipVpZuWdpFTQAttR18hy1K1BbokNRX4R1bLtzTi, address:104.36.82.53:4122): protocol: write tcp 188.166.225.170:44694->104.36.82.53:4122: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:06:19.254792147 (85h30m28.977355187s) [seg_pos={0 0}, upload=82020]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 01:06:19.662258226 (85h30m29.384821266s) [upload=82020]: Done uploading segments... Jul 28 01:06:19.662282563 (85h30m29.384845603s) [upload=82020]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.410181882s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 1.935950184s FAIL (2025-07-28 01:06:13 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.935975211s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.....X.X...........X...................X.................] duration: 5.892579162s FAIL (2025-07-28 01:06:09 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.892603249s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 1.951075534s FAIL (2025-07-28 01:05:41 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.951096685s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.....X.X...........X...................X................] duration: 5.860671795s FAIL (2025-07-28 01:05:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== BFN6Y7NF7VR3H3T75CH67NRUACPHZA, seg_pos={0 0}, upload=81806]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX): protocol: context canceled; context canceled Jul 28 00:51:17.036549435 (85h15m26.759112555s) [seg_pos={0 0}, upload=81806]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:51:17.036704020 (85h15m26.75926708s) [upload=81806]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:51:17.036712447 (85h15m26.759275547s) [upload=81806]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:51:17.316822538 (85h15m27.039385588s) [upload=81806]: Done uploading segments... Jul 28 00:51:17.316848307 (85h15m27.039411357s) [upload=81806]: Upload finished. err: <nil> Jul 28 00:52:24.217040202 (85h16m33.939603242s) [upload=81822]: Starting upload Jul 28 00:52:24.217107020 (85h16m33.93967006s) [upload=81822]: Done starting upload Jul 28 00:52:24.217197382 (85h16m33.939760422s) [upload=81822]: Uploading segments... Jul 28 00:52:24.217396484 (85h16m33.939959524s) [upload=81822]: Got next segment. Inline: false Jul 28 00:52:24.217448233 (85h16m33.940011263s) [upload=81822]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:52:24.217453262 (85h16m33.940016302s) [upload=81822]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:52:24.468973282 (85h16m34.191536322s) [seg_pos={0 0}, upload=81822]: Begin upload segment... Jul 28 00:52:24.469017267 (85h16m34.191580307s) [seg_pos={0 0}, upload=81822]: Done begin upload segment. Jul 28 00:52:24.469037415 (85h16m34.191600445s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469381974 (85h16m34.191945014s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469631622 (85h16m34.192194662s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.469897411 (85h16m34.192460451s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470153050 (85h16m34.19271609s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470410482 (85h16m34.192973522s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470649830 (85h16m34.19321287s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.470836978 (85h16m34.193400018s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471018996 (85h16m34.193582036s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471209571 (85h16m34.193772611s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471398071 (85h16m34.193961101s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471585751 (85h16m34.194148791s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471763771 (85h16m34.194326811s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.471946200 (85h16m34.19450924s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472124320 (85h16m34.194687361s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:24.472317059 (85h16m34.194880099s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Uploading piece... Jul 28 00:52:26.218563198 (85h16m35.941126268s) [upload=81822]: Next returned nil segment Jul 28 00:52:26.218582566 (85h16m35.941145626s) [upload=81822]: Waiting for error group managing segments... Jul 28 00:52:26.506105560 (85h16m36.2286686s) [noise=true , address=45.91.116.4:4937 , piece_id=6N6MGWFZABLS7VPHRBSLJX7FP5UP2L52YOQI222V4F7ZXAZJY5PA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.512788065 (85h16m36.235351095s) [noise=true , address=77.90.31.16:4733 , piece_id=6HBOO4F5YIC5IFKSOCMRTEYVT6PN3JF4VNLBALNR5MLVB43QOB4Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.543356551 (85h16m36.265919591s) [noise=true , address=45.91.116.6:4975 , piece_id=OC32BA36EVRV5LUWVPTV5ZXIYS3HIXIUIKAZ2JA3O7VCVEI4NF3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.566429095 (85h16m36.288992136s) [noise=true , address=77.90.31.15:4698 , piece_id=F5JVMEWHHFCVYG2EGLHJL2UWJ5Z6QKK62QGMEDLLMEB7RMZBZSTA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.569915646 (85h16m36.292478726s) [noise=true , address=45.91.116.12:4225 , piece_id=L7YF7M2LWAHJ7MUFZRVHGQ27X5UBPHPLIN2PONIPQBYH5AB5D74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.576667414 (85h16m36.299230495s) [noise=true , address=77.90.31.10:4518 , piece_id=O2VN3LDRSF2K6HOQAWAP55Z7J66JCQZAU6IX2HYRBSE7XPEIM32Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.585264903 (85h16m36.307827943s) [noise=true , address=77.90.31.14:4662 , piece_id=3DWZIMOB4VUMDTZ4ZGMJH7ZHEQ3YEKAHRA7QFADBHLFQDCM5M63Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.588048861 (85h16m36.310611901s) [noise=true , address=45.91.116.5:4298 , piece_id=ZSSYSKWCCQMZYYHSBMJADD462BTKARKW3GOYCHMFPYQWBW6Z2MQQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.609178818 (85h16m36.331741858s) [noise=true , address=45.91.116.13:4155 , piece_id=DWYTUFR54B66JW3LPH2OCBRPHQAAU3DURQPUTUXSFHMYHCWTNCHA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.709346698 (85h16m36.431909738s) [noise=true , address=77.90.31.11:4553 , piece_id=CV5KW4SLYXNC4AN4UEZXVH5VU6OPABUDUGD3YYAS7N6YOFF7P74Q, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.749164100 (85h16m36.47172713s) [noise=true , address=45.91.116.11:4118 , piece_id=G7FOYCT6UM75QSXIL3AUPRVU2ZRZRNEP52OI75YTMK6KG7UG3R3A, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:26.764501434 (85h16m36.487064474s) [noise=true , address=77.90.31.21:4879 , piece_id=7LKTIUQX3XTE4U44DNBRKLTDJ3OUBRCDIQ6I7MQBHUSHKYKZBQCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: <nil> Jul 28 00:52:53.577899248 (85h17m3.300462288s) [noise=true , address=104.36.82.3:4765 , piece_id=2ECZXZC2FWUER2ZGC27OHZXHS2TVMEDVIE5ALHV5BDAMYV6PJ5ZQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: write tcp 188.166.225.170:55240->104.36.82.3:4765: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577922302 (85h17m3.300485342s) [noise=true , address=104.36.82.6:4876 , piece_id=HNOQTZICTCXHWWQDZUGY7YBJT7RZSRDU5JUN2AUASS6G2M3BM6EQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: write tcp 188.166.225.170:33644->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577950035 (85h17m3.300513075s) [noise=true , address=104.36.82.2:4729 , piece_id=HR3S6KR6DDCPJQCNPPHFDY7NZPERTUGYLZGQZE4XNTRL3KJIYDCQ, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: write tcp 188.166.225.170:44878->104.36.82.2:4729: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577975404 (85h17m3.300538444s) [noise=true , address=104.36.82.6:4874 , piece_id=3P5UNY537XZGGJBFZB7TBXRGXIYCM2L6WRHNSH4QG34L5NCOPBFA, seg_pos={0 0}, upload=81822]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 188.166.225.170:60094->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:52:53.577993918 (85h17m3.300556959s) [seg_pos={0 0}, upload=81822]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:52:53.808803314 (85h17m3.531366354s) [upload=81822]: Done uploading segments... Jul 28 00:52:53.808835125 (85h17m3.531398165s) [upload=81822]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.860700328s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [XX....X....X.......X.....X...............................X..] duration: 5.715398145s FAIL (2025-07-28 01:05:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.715430628s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 56 out of 60 is failed (threshold=3) [.X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.235225526s FAIL (2025-07-28 01:05:19 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== UXEYK6SEDRQ, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.894543654 (85h28m29.617106694s) [noise=true , address=104.36.82.12:4182 , piece_id=IFKI6LTDMBNKB26N7IWWEIFQXGPIHKNTFIHH3RQNCUEDZCXBPKUQ, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.894734008 (85h28m29.617297048s) [noise=true , address=104.36.82.11:4141 , piece_id=LV3ENYUI5D6V5BREA4BTKDFV2GBPFVD5MLBKETYFK54BGL65VGZA, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.894739629 (85h28m29.617302669s) [noise=true , address=104.36.82.54:4168 , piece_id=TTC7J4UEJYT642JEFGYU7T6X22QMAII7HOODHCQLDZ4D42LB6FDA, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.894770508 (85h28m29.617333548s) [noise=true , address=104.36.82.55:4205 , piece_id=PBOJHHFYDG73E325P556BO4NBNNGAWVVLNBRZXSQG67KL3ESL5PQ, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.895177637 (85h28m29.617740677s) [noise=true , address=104.36.82.15:4283 , piece_id=4FFE3TA7GCFYNWLTRKMS254QHF4KQYHITB5P66GHJT5PU7JUZS4A, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:19.895446201 (85h28m29.618009231s) [noise=true , address=104.36.82.13:4219 , piece_id=E7KOWKC4SFUGIGMP3KFWEUUXIBXAJHWEQ4TQYJ6P2CERQKF4NNHA, seg_pos={0 0}, upload=82003]: Uploading piece... Jul 28 01:04:20.736473386 (85h28m30.459036426s) [noise=true , address=64.49.13.138:4482 , piece_id=ZPQR2PSMSGUHJGHPGH37BZ2MQFWDICAWXU3TF3TMCGTXK2ITFSPA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 28 01:05:18.786629159 (85h29m28.509192199s) [upload=82003]: Next segment err: context deadline exceeded Jul 28 01:05:18.786951727 (85h29m28.509514767s) [noise=true , address=104.36.82.22:4542 , piece_id=DD4SRVTBFP2X7MRA4ZF7F3FDSWFAMWWDUGEHKKZBGMUURXEB6TYA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1V1MxSKuV5ZxsXXJBppZ5qmFwJPovFp8wD1biE1sF7mWzQ9qpL, address:104.36.82.22:4542): protocol: write tcp 188.166.225.170:38754->104.36.82.22:4542: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.786994037 (85h29m28.509557077s) [noise=true , address=104.36.82.3:4793 , piece_id=DIENHTAG7A7MTUGQAMX657VV7WOZD22SA5TPQVKJ65QMBYYK3IRA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12CffFfbiSKKBGuksEnNL1NuVRjbkRdVUjoYMJisvUM6pEiaUgJ, address:104.36.82.3:4793): protocol: write tcp 188.166.225.170:53628->104.36.82.3:4793: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787071646 (85h29m28.509634676s) [noise=true , address=104.36.82.50:4018 , piece_id=7LJFEDINNE4HEWK3WHK2P6YWDF5YS22GUHODAWVQ2OKQSEBGU6IQ, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12YcYgDWq2k7F9fwpdLAGNkrskGEUURe6QuRed7mRjMTdNHEQ8D, address:104.36.82.50:4018): protocol: rpc: dial tcp 104.36.82.50:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787164504 (85h29m28.509727544s) [noise=true , address=104.36.82.12:4182 , piece_id=IFKI6LTDMBNKB26N7IWWEIFQXGPIHKNTFIHH3RQNCUEDZCXBPKUQ, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1KSPtVhJPKYAb3Gj25ihyExNgkDCGo9QTDaFcEfXRfLdDz8nsh, address:104.36.82.12:4182): protocol: rpc: dial tcp 104.36.82.12:4182: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787218888 (85h29m28.509781928s) [noise=true , address=104.36.82.14:4229 , piece_id=QBJHRWA5H5ENOY7ADZPPUDVLE2PAKAEC3QZJY5PSLXA6VWNBM7VA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12uiCMvJ7pnwXfN57CVu8zR73tikwUZiXr3sG7Xotd9w6KVnViL, address:104.36.82.14:4229): protocol: write tcp 188.166.225.170:35900->104.36.82.14:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787253354 (85h29m28.509816394s) [noise=true , address=104.36.82.3:4785 , piece_id=TLCDXKJL5SQXQFJOP7MSK2MPT324BTZVW2W6MD2ZSNFOZRKJOHHA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12u5zY7F7oy2tPUyFfoiebzFEySetBks9LC3gRGPURAyDDnsUu9, address:104.36.82.3:4785): protocol: write tcp 188.166.225.170:37172->104.36.82.3:4785: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787283151 (85h29m28.509846191s) [noise=true , address=104.36.82.42:4811 , piece_id=BLUHRB67JLFOIHBIT3BHNOAGH56PPWXVR45RJWPKWVXIQTK35YYA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:124LaFQsBrTNvKgaKTQRHtHxVH6jab6CMTdaRWs7HhfmmYvpef3, address:104.36.82.42:4811): protocol: write tcp 188.166.225.170:36780->104.36.82.42:4811: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787312677 (85h29m28.509875717s) [noise=true , address=104.36.82.51:4049 , piece_id=OOIVE7GPXVRZW7Y7V6QMOM5PHAVU4VBF4JBHSZVPXICZKMVH42JQ, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J, address:104.36.82.51:4049): protocol: write tcp 188.166.225.170:43010->104.36.82.51:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787345289 (85h29m28.50990834s) [noise=true , address=104.36.82.55:4205 , piece_id=PBOJHHFYDG73E325P556BO4NBNNGAWVVLNBRZXSQG67KL3ESL5PQ, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12UafxBVo9nuRVBcQmGE4gBP2pnXQ5YUrwHZdwPCeSBSBWCZKpQ, address:104.36.82.55:4205): protocol: rpc: dial tcp 104.36.82.55:4205: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787381990 (85h29m28.50994502s) [noise=true , address=104.36.82.56:4237 , piece_id=CD7WXPUSDPWD4OLYU6LZVVPBPSRJ5K6YPY3UL56R7HEKHRL6W3PA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1VzewCRbNpJv7kNwp9wruQuoVSaZFjA44qN33Ct7qm1GnfhtDj, address:104.36.82.56:4237): protocol: write tcp 188.166.225.170:43614->104.36.82.56:4237: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787418249 (85h29m28.509981289s) [noise=true , address=104.36.82.15:4283 , piece_id=4FFE3TA7GCFYNWLTRKMS254QHF4KQYHITB5P66GHJT5PU7JUZS4A, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1ytKjZnmERiZtgJuMJhXG5KFwUQwie27odve4LPE7Pw5hR5ZaU, address:104.36.82.15:4283): protocol: write tcp 188.166.225.170:46480->104.36.82.15:4283: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787444770 (85h29m28.51000781s) [noise=true , address=104.36.82.53:4136 , piece_id=BVFZBBSUGZWZ263XZPSYCL3QB6KSLSUWEQGBZ7UQVISJURYTICJA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12rkSxE1mCbyoUyh3pPwz6xdw8rXqpeyFrhciFoB8bgs4TkKmyT, address:104.36.82.53:4136): protocol: write tcp 188.166.225.170:36772->104.36.82.53:4136: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787457765 (85h29m28.510020805s) [noise=true , address=104.36.82.46:4500 , piece_id=O2I5PDE6LHO7O5AYKO5TRK2VF32GP2UOK2Z6457KMUXEYK6SEDRQ, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12B7ox4ehqxJ2mwqeXj8C9N9GuSU5etNC23iT1pJnTgMLGBAGQK, address:104.36.82.46:4500): protocol: write tcp 188.166.225.170:46616->104.36.82.46:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787481149 (85h29m28.510044189s) [noise=true , address=104.36.82.10:4109 , piece_id=XBAKDZBV4D3IM2LIZWETNBFBNNZIH3FEO333MFWGT4X7PRH4RTXA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12ZE1ut9Tum4tzq2m8zbFaGUNLdNZSZQYARkayDAfE1b7uZtWjW, address:104.36.82.10:4109): protocol: write tcp 188.166.225.170:55560->104.36.82.10:4109: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787507179 (85h29m28.510070219s) [noise=true , address=104.36.82.11:4141 , piece_id=LV3ENYUI5D6V5BREA4BTKDFV2GBPFVD5MLBKETYFK54BGL65VGZA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:129zfpWSguxcUH9723ivdoL39WgpjiTxA189mYbzsGAGVjFcXLL, address:104.36.82.11:4141): protocol: write tcp 188.166.225.170:35946->104.36.82.11:4141: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787514212 (85h29m28.510077243s) [noise=true , address=104.36.82.13:4219 , piece_id=E7KOWKC4SFUGIGMP3KFWEUUXIBXAJHWEQ4TQYJ6P2CERQKF4NNHA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:12fUzLg93yKvyKkNvuvcKZHyvJ2HGE6tUCg1Wm7ZZjKKeDZDWm9, address:104.36.82.13:4219): protocol: write tcp 188.166.225.170:56578->104.36.82.13:4219: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787544991 (85h29m28.510108031s) [noise=true , address=104.36.82.54:4168 , piece_id=TTC7J4UEJYT642JEFGYU7T6X22QMAII7HOODHCQLDZ4D42LB6FDA, seg_pos={0 0}, upload=82003]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: write tcp 188.166.225.170:34280->104.36.82.54:4168: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:05:18.787566753 (85h29m28.510129783s) [seg_pos={0 0}, upload=82003]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.235239653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 1.959705372s FAIL (2025-07-28 01:05:03 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.959734929s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 1.898925715s FAIL (2025-07-28 01:04:41 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.898949702s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.XX....X....X.......X.....X...............................X.] duration: 5.744707733s FAIL (2025-07-28 01:04:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.744731478s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)