Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.X..........X.....X.X...........X................] duration: 5.908267016s FAIL (2025-07-28 00:49:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.90829576s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [.................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234542888s FAIL (2025-07-28 00:48:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== P3DUOCOUBOFAPHJBRVA7AAZIRURKHDPPVZXUA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824492279 (85h12m0.547055319s) [noise=true , address=104.36.82.41:4780 , piece_id=2WQEH7BT72ZLUICMUZKBXZW67WB6MV6PIC4IYUKAVIDZZZFD62GA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824528007 (85h12m0.547091047s) [noise=true , address=79.127.201.215:4357 , piece_id=MTA4CHPP4ONRXJHYO4HENNFAJVDUBSLOV7CZEKDABU23F6OBQ32A, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824729573 (85h12m0.547292613s) [noise=true , address=104.36.82.55:4196 , piece_id=XDII2ZSEQTM55BJ35S5J7XPBSHD4MGZJ2PCMMPFSBVR74W5U66BA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.824767195 (85h12m0.547330235s) [noise=true , address=104.36.82.53:4148 , piece_id=C7THPNW2BGNINLZVA2MICVENCSE7LLJHLVRSHAC5LNSUTLTN5NLA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825034997 (85h12m0.547598037s) [noise=true , address=104.36.82.14:4231 , piece_id=MIXFIEDT6LBNSBJKKVLFIHD3O3OMKBUUPHRBKSPJW3VV4AVTRCLA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825323740 (85h12m0.54788678s) [noise=true , address=104.36.82.16:4314 , piece_id=ZWLE4A6WS4FLDF35WAWZ3C4OFYSKRVHXW2J2S5HWT7BOC6W6UUWA, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:50.825637760 (85h12m0.5482008s) [noise=true , address=104.36.82.42:4828 , piece_id=RVLHJ2QHKZRKRUODM7ATS66NC4Y6PKQ7DWRYM6F2YNAZSME2Q3DQ, seg_pos={0 0}, upload=81751]: Uploading piece... Jul 28 00:47:51.937028269 (85h12m1.659591299s) [upload=81751]: Next returned nil segment Jul 28 00:47:51.937047556 (85h12m1.659610596s) [upload=81751]: Waiting for error group managing segments... Jul 28 00:47:52.309093908 (85h12m2.031656938s) [noise=true , address=79.127.205.234:4618 , piece_id=BREWR5ZI2WEGSYX7A4VK23PGFSGPWCOIQDP24I7KNUDSGUJI3TJQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: <nil> Jul 28 00:47:57.179542618 (85h12m6.902105658s) [noise=true , address=79.127.201.215:4357 , piece_id=MTA4CHPP4ONRXJHYO4HENNFAJVDUBSLOV7CZEKDABU23F6OBQ32A, seg_pos={0 0}, upload=81751]: Done uploading piece. err: <nil> Jul 28 00:48:49.729868761 (85h12m59.452431801s) [noise=true , address=104.36.82.15:4292 , piece_id=WVRCUVLAUYXXGZILE6WO3H45YSM2ZNN5MBJPK2V4DIFP4EAAOKRQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:104.36.82.15:4292): protocol: rpc: dial tcp 104.36.82.15:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.729923726 (85h12m59.452486766s) [noise=true , address=104.36.82.41:4780 , piece_id=2WQEH7BT72ZLUICMUZKBXZW67WB6MV6PIC4IYUKAVIDZZZFD62GA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:104.36.82.41:4780): protocol: rpc: dial tcp 104.36.82.41:4780: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.729969383 (85h12m59.452532423s) [noise=true , address=104.36.82.14:4231 , piece_id=MIXFIEDT6LBNSBJKKVLFIHD3O3OMKBUUPHRBKSPJW3VV4AVTRCLA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ixY5pokE6MtrVH9FpKc1tGZhEAaW5brbzBeeGmYUczK3z7tCQ, address:104.36.82.14:4231): protocol: rpc: dial tcp 104.36.82.14:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730004020 (85h12m59.45256706s) [noise=true , address=104.36.82.16:4314 , piece_id=ZWLE4A6WS4FLDF35WAWZ3C4OFYSKRVHXW2J2S5HWT7BOC6W6UUWA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12KjL9YES3swMrgFERTQczzsc62AcXVJXC3sFuHsSt5M4DHt5Xw, address:104.36.82.16:4314): protocol: rpc: dial tcp 104.36.82.16:4314: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730062552 (85h12m59.452625592s) [noise=true , address=104.36.82.4:4811 , piece_id=CHYYWFJCJZAU4FGINEMLDC6XLTSRLFLXIQFALIY3EQRFBAKRIZBA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V, address:104.36.82.4:4811): protocol: rpc: dial tcp 104.36.82.4:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730096386 (85h12m59.452659426s) [noise=true , address=104.36.82.51:4073 , piece_id=CMHJN7GO6PDDNKEHCXNCBWPZC75LBXMSFDC4CHNXCOFT22W3B6YA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1kGo5nNz8GyroAGoMQ75xDo4pppWPVsW9f8trLCWH6CqVXWsmK, address:104.36.82.51:4073): protocol: rpc: dial tcp 104.36.82.51:4073: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730136142 (85h12m59.452699182s) [noise=true , address=104.36.82.42:4828 , piece_id=RVLHJ2QHKZRKRUODM7ATS66NC4Y6PKQ7DWRYM6F2YNAZSME2Q3DQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1cX7FtKC8KMopnX9zvkT6fYdnutK84cFjLW6xMwpXuGsqpiN8k, address:104.36.82.42:4828): protocol: write tcp 188.166.225.170:60530->104.36.82.42:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730162873 (85h12m59.452725913s) [noise=true , address=104.36.82.55:4196 , piece_id=XDII2ZSEQTM55BJ35S5J7XPBSHD4MGZJ2PCMMPFSBVR74W5U66BA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12Vs4grpszw7MhLJSMgCn7ikaL4jgwSB6XULfVgAt1Msy8hMcaL, address:104.36.82.55:4196): protocol: write tcp 188.166.225.170:38440->104.36.82.55:4196: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730183212 (85h12m59.452746252s) [noise=true , address=104.36.82.54:4157 , piece_id=XKMAT7RSKKGK5EFMEWOF7XCZGXKASV7ZTNVKBTEPQIJAVSSA7JEA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: write tcp 188.166.225.170:57592->104.36.82.54:4157: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730219311 (85h12m59.452782351s) [noise=true , address=104.36.82.52:4085 , piece_id=4Z6OBA2HHBORS55P3DUOCOUBOFAPHJBRVA7AAZIRURKHDPPVZXUA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C, address:104.36.82.52:4085): protocol: write tcp 188.166.225.170:45628->104.36.82.52:4085: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730223189 (85h12m59.452786229s) [noise=true , address=104.36.82.10:4108 , piece_id=CW3NEQE25H7QFR3VZJTXZYMT6UDL4TZKPRJCN3HHMQ4QXFSOMTXA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1ApRJLbSnYn9Bc1t1VN2UL1wLzWJ2pBmZG1BKidJBD3do4T4p1, address:104.36.82.10:4108): protocol: write tcp 188.166.225.170:45116->104.36.82.10:4108: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730255000 (85h12m59.45281804s) [noise=true , address=104.36.82.53:4148 , piece_id=C7THPNW2BGNINLZVA2MICVENCSE7LLJHLVRSHAC5LNSUTLTN5NLA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: write tcp 188.166.225.170:35014->104.36.82.53:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730274556 (85h12m59.452837596s) [noise=true , address=104.36.82.3:4796 , piece_id=RHINQ3DLTDF4HFWX3AW2XKXPAKWW52YLTWSUO3Z4CXUPPJNB2UQQ, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12nZwQ4Poz9M14ijFV6qpy5SCWEMeyHQpxN17uSBTH1ts1CFtVS, address:104.36.82.3:4796): protocol: write tcp 188.166.225.170:56424->104.36.82.3:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730291439 (85h12m59.452854479s) [noise=true , address=104.36.82.12:4164 , piece_id=TD7PDB2M3RFF2PVKRJOPWEN5BWO2LPJMY2T7PMZ4YHQJ2ZP7STJA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:12iQgyDiz9vu1TbCRuATGmnrfbgBaia72b77ZjQJ6gpnEFbQzTJ, address:104.36.82.12:4164): protocol: write tcp 188.166.225.170:47366->104.36.82.12:4164: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730318511 (85h12m59.452881551s) [noise=true , address=104.36.82.52:4088 , piece_id=HB2Y42AYUKGBVWO2LIISIQQJNSK5ICMH7FKWVYAFB4BPV6UTUJJA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1284h6cPNVXJYMkSekGQVw8b7p89tkhpmgYLxTTYUDu2J7Wadoj, address:104.36.82.52:4088): protocol: write tcp 188.166.225.170:49576->104.36.82.52:4088: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730340954 (85h12m59.452903994s) [noise=true , address=104.36.82.6:4884 , piece_id=V4KZADRDOGCKHOOW3LOY3SXYNSAJ5RFU7YGQMPQGVSW45Q7XJOPA, seg_pos={0 0}, upload=81751]: Done uploading piece. err: ecclient: upload failed (node:1c4THqUrQMeqY94EGiZSNKQDDnvycEZbGMrYxrb4s4XuaRe8br, address:104.36.82.6:4884): protocol: write tcp 188.166.225.170:53340->104.36.82.6:4884: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:48:49.730368457 (85h12m59.452931497s) [seg_pos={0 0}, upload=81751]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:48:49.962194256 (85h12m59.684757296s) [upload=81751]: Done uploading segments... Jul 28 00:48:49.962221167 (85h12m59.684784207s) [upload=81751]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234567635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.......X.XX....X....X.......X.....X............] duration: 5.84133658s FAIL (2025-07-28 00:48:49 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.841364063s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.X..........X.....X.X...........X...............] duration: 5.933898055s FAIL (2025-07-28 00:48:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.93392711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.......................X.........................] duration: 1.869117996s FAIL (2025-07-28 00:48:24 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.869141481s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.......X.XX....X....X.......X.....X...........] duration: 5.718897998s FAIL (2025-07-28 00:47:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.71892449s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230436514s FAIL (2025-07-28 00:47:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== ise=true , address=104.36.82.11:4133 , piece_id=Y5CW4L2NQ6GNPQZVMO4HPTCN5XORNIL5MI7ARL2WTIC3BEN7QE2Q, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308356486 (85h10m59.030919526s) [noise=true , address=104.36.82.54:4189 , piece_id=GAQOZLDW3FHCXSDW3VH3LNDDHY5QL3DGKVR7I2U7NJBI563IXKTQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308683772 (85h10m59.031246812s) [noise=true , address=104.36.82.15:4281 , piece_id=GFGYW2UH6YNG5JQ2JDCUVEJCSDZ4FD6Z7Z54YJQHGIONKKYB5FEA, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.308977224 (85h10m59.031540264s) [noise=true , address=104.36.82.14:4228 , piece_id=J2KFJJHU3GNOKO3MRDUM2XN7BVSIHXW7YNPD5TQOMT52ZSCYNNJQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.309605155 (85h10m59.032168195s) [noise=true , address=104.36.82.10:4096 , piece_id=H7JKLD5TCNLLNAXQ4A4CPBTGAJK3Z5VPM3KNEXZV5IYXZLIUSURQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.310038013 (85h10m59.032601053s) [noise=true , address=104.36.82.9:4033 , piece_id=V6SY33FJO4THYNVFD3HM7JIDLE5WTWKNXWXRHVTKMZF2OYTJ5ITA, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.310714738 (85h10m59.033277778s) [noise=true , address=104.36.82.13:4203 , piece_id=DBWSH7AIGUARAP5AIGOPMGLAXO2EOM7ZABT7QE73TYPTXDMM7MDQ, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.311061241 (85h10m59.033624281s) [noise=true , address=104.36.82.41:4778 , piece_id=AQ77RCBAKM3YMAF4O4DHH5WWKE2DZDU3WCF5F36IX6NH52BJUF6A, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:49.311654055 (85h10m59.034217095s) [noise=true , address=104.36.82.40:4965 , piece_id=MWA6HELA4R7DEWRXVVYYX2HKOXKHIJIUGEFNAR5FYLMCFELVCH2A, seg_pos={0 0}, upload=81736]: Uploading piece... Jul 28 00:46:50.816271418 (85h11m0.538834459s) [upload=81736]: Next returned nil segment Jul 28 00:46:50.816291046 (85h11m0.538854086s) [upload=81736]: Waiting for error group managing segments... Jul 28 00:46:54.564772504 (85h11m4.287335584s) [noise=true , address=64.49.13.138:4483 , piece_id=OPQIO74C5MWPYGABCR2KIEAGHGR35AYRYCAW4DI7FFY2XKQC4XRA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: <nil> Jul 28 00:46:59.804875411 (85h11m9.527438461s) [noise=true , address=79.127.201.215:4349 , piece_id=U4JOTVOB2IUI2CCNNBQR7JDBLTLB4QVKS6XIJNBILGT2T7EB4POA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: <nil> Jul 28 00:47:48.218395891 (85h11m57.940958931s) [noise=true , address=104.36.82.52:4102 , piece_id=GG7FMXCSVVSIII2FKBYCF2WBW6D4VI2EZ3WUFMNCZL52TRFOVNBQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12evG7WqJ9XXqh2z2kjGr4Wz1q5mrcySfTMczqEm7i8jgYsW8qP, address:104.36.82.52:4102): protocol: rpc: dial tcp 104.36.82.52:4102: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218455796 (85h11m57.941018836s) [noise=true , address=104.36.82.42:4803 , piece_id=2GOY3TITYM2OI7NLDTU35LRNIVV5IOC2ATXXSSQYPXWF664CTO2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12GFk3GkhB88UKQdZM8vZ9VLNuuYDrS5ThZjVAdbL2CtTQVMDEB, address:104.36.82.42:4803): protocol: rpc: dial tcp 104.36.82.42:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218488187 (85h11m57.941051217s) [noise=true , address=104.36.82.11:4133 , piece_id=Y5CW4L2NQ6GNPQZVMO4HPTCN5XORNIL5MI7ARL2WTIC3BEN7QE2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: rpc: dial tcp 104.36.82.11:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218509789 (85h11m57.941072829s) [noise=true , address=104.36.82.9:4033 , piece_id=V6SY33FJO4THYNVFD3HM7JIDLE5WTWKNXWXRHVTKMZF2OYTJ5ITA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1SV3LrcQW16wLewDzfyUWFB6sUgnERNJVdhz72CJmyde95M79C, address:104.36.82.9:4033): protocol: rpc: dial tcp 104.36.82.9:4033: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218530509 (85h11m57.941093539s) [noise=true , address=104.36.82.15:4281 , piece_id=GFGYW2UH6YNG5JQ2JDCUVEJCSDZ4FD6Z7Z54YJQHGIONKKYB5FEA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12hDXeAm4tomp1gAkBNjM8x77V76yiruYQmPQ3BZEYPLpJzigo2, address:104.36.82.15:4281): protocol: rpc: dial tcp 104.36.82.15:4281: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218562790 (85h11m57.941125821s) [noise=true , address=104.36.82.41:4778 , piece_id=AQ77RCBAKM3YMAF4O4DHH5WWKE2DZDU3WCF5F36IX6NH52BJUF6A, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV, address:104.36.82.41:4778): protocol: rpc: dial tcp 104.36.82.41:4778: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218587548 (85h11m57.941150588s) [noise=true , address=104.36.82.13:4203 , piece_id=DBWSH7AIGUARAP5AIGOPMGLAXO2EOM7ZABT7QE73TYPTXDMM7MDQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:18cNSsgqn3n6z5LxRcVCRzDzfFDpvptV1MKh6NdNRLW4bg9fWM, address:104.36.82.13:4203): protocol: rpc: dial tcp 104.36.82.13:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218611113 (85h11m57.941174153s) [noise=true , address=104.36.82.44:4869 , piece_id=4WCILWRWHGQA4OWYF4HDKX6U6NQ7ETTVBH52MWQQCZR4DGUY6TKQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq, address:104.36.82.44:4869): protocol: rpc: dial tcp 104.36.82.44:4869: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218635310 (85h11m57.94119835s) [noise=true , address=104.36.82.54:4189 , piece_id=GAQOZLDW3FHCXSDW3VH3LNDDHY5QL3DGKVR7I2U7NJBI563IXKTQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: rpc: dial tcp 104.36.82.54:4189: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218655768 (85h11m57.941218808s) [noise=true , address=104.36.82.6:4899 , piece_id=UJ3QCGBRRVD7QHXC6BICPXHTH6XCKM4BAXQ4CJKCWRJ5NG2S6K2Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1tdNNPMJKCqJ39U1f4aj3Xrwh7J12jpyQGoQDQUdWNCxszcA5k, address:104.36.82.6:4899): protocol: rpc: dial tcp 104.36.82.6:4899: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218678031 (85h11m57.941241071s) [noise=true , address=104.36.82.14:4228 , piece_id=J2KFJJHU3GNOKO3MRDUM2XN7BVSIHXW7YNPD5TQOMT52ZSCYNNJQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12pkADNFaYa2SqFALQdxwf1CfR1ktUwshQkw7iBJzhdhaRYaszF, address:104.36.82.14:4228): protocol: rpc: dial tcp 104.36.82.14:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218699362 (85h11m57.941262402s) [noise=true , address=104.36.82.55:4203 , piece_id=BZ3PQU4AZA5CILSNRVVTEGUF2MO2YV275JGWN4KSHYBDYIF3FIFQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: rpc: dial tcp 104.36.82.55:4203: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218712697 (85h11m57.941275737s) [noise=true , address=104.36.82.10:4096 , piece_id=H7JKLD5TCNLLNAXQ4A4CPBTGAJK3Z5VPM3KNEXZV5IYXZLIUSURQ, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1U9C1EMFMyt1sdCP5pEhckjj7fW75S7vUbswchN1Rp4s3b6RkA, address:104.36.82.10:4096): protocol: rpc: dial tcp 104.36.82.10:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218742955 (85h11m57.941305985s) [noise=true , address=104.36.82.5:4866 , piece_id=VKKZW44M7CKZJEKJBSEPMH373H4TLWWXVUCYJYFXHXWBQJO3IILA, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1zUcYfJE8xRW2Tow7bGB3PeNrVBUzVWR82JADkVwPdSr4CTPyr, address:104.36.82.5:4866): protocol: write tcp 188.166.225.170:48178->104.36.82.5:4866: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218756922 (85h11m57.941319962s) [noise=true , address=104.36.82.40:4965 , piece_id=MWA6HELA4R7DEWRXVVYYX2HKOXKHIJIUGEFNAR5FYLMCFELVCH2A, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1ToFTqjRAWvpRQzf6vyHKDpforPyb4LbDGcCVhRhkwiBGKihnD, address:104.36.82.40:4965): protocol: write tcp 188.166.225.170:46974->104.36.82.40:4965: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218766750 (85h11m57.94132979s) [noise=true , address=104.36.82.12:4159 , piece_id=RXACT2D2ORFZ4OCFYT73WH7YTJ7HKOK7WR3YQ6G7HNIS37XXDF5Q, seg_pos={0 0}, upload=81736]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: write tcp 188.166.225.170:58666->104.36.82.12:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:47:48.218789254 (85h11m57.941352294s) [seg_pos={0 0}, upload=81736]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:47:48.447223334 (85h11m58.169786374s) [upload=81736]: Done uploading segments... Jul 28 00:47:48.447248201 (85h11m58.169811241s) [upload=81736]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.23045018s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............X.X..........X.....X.X...........X..............] duration: 5.804084816s FAIL (2025-07-28 00:47:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.804111419s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X.......................X........................] duration: 1.868194931s FAIL (2025-07-28 00:47:22 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.868228074s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.......X.XX....X....X.......X.....X..........] duration: 5.741435618s FAIL (2025-07-28 00:47:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.741467992s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.............X.X..........X.....X.X...........X.............] duration: 5.875800167s FAIL (2025-07-28 00:46:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.875830334s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [...................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232790014s FAIL (2025-07-28 00:46:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== [noise=true , address=104.36.82.42:4821 , piece_id=PGVY7G42RSSXOMLBYF65H6THBBAEPG74WR7JACAWNGAHAG3OQ7NQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810365758 (85h9m58.532928798s) [noise=true , address=104.36.82.40:4971 , piece_id=DYFZ7K7DEX5OYIR7A2HWBTXDCPI6Y5BBXEWBC6S5OB3WHRRGXZJQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810599745 (85h9m58.533162775s) [noise=true , address=104.36.82.49:4992 , piece_id=VOHZXDUQJWHPEVIET3LQUMYQIFWQUP75UDUA4AVCF7J4UO3VK5JA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.810818273 (85h9m58.533381313s) [noise=true , address=104.36.82.14:4242 , piece_id=DPAD7IVXUNT3LRHWKNOOIA72QRRWBLE5FW7D4A2HOGZ6HUG6OKQA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811031561 (85h9m58.533594601s) [noise=true , address=104.36.82.13:4194 , piece_id=O54LDQ5TUJ7G7NC7ZPXVFLS4RNHQX5SLRSL3FHNODD5UGOQA7WSQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811222246 (85h9m58.533785286s) [noise=true , address=104.36.82.54:4179 , piece_id=FPUMX7NO22OSSSXCYBFRNFKU67HP4IOG6ENHY45F3EGHAKPEXBBQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811432288 (85h9m58.533995318s) [noise=true , address=104.36.82.56:4235 , piece_id=VYXBYYD4UEP65672CE2HZ3LLJ6SKBXUYXDQGARIVTQFNU4Y4W2FQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811630728 (85h9m58.534193768s) [noise=true , address=104.36.82.41:4773 , piece_id=PTXD3EUPC7R2M4J3V47B6W4TXEV6VSB66HU5XFZKJDV5E2DFXURQ, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:48.811837633 (85h9m58.534400673s) [noise=true , address=104.36.82.11:4145 , piece_id=DS7DX6FKGY4YUCHSJAYCPN5WTUPX3CBK4X3TUSOMDXWVTWRH7GEA, seg_pos={0 0}, upload=81725]: Uploading piece... Jul 28 00:45:49.923978246 (85h9m59.646541287s) [upload=81725]: Next returned nil segment Jul 28 00:45:49.923997843 (85h9m59.646560924s) [upload=81725]: Waiting for error group managing segments... Jul 28 00:45:50.309514087 (85h10m0.032077157s) [noise=true , address=79.127.205.235:4688 , piece_id=PXHJ3DCV3PPOPV44TKZSWEBN5DHBVCQ3YADPOAD7YHJVNJ55SSEA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: <nil> Jul 28 00:45:50.549932185 (85h10m0.272495235s) [noise=true , address=79.127.220.97:4375 , piece_id=Z7KYI2LTSJ3LGBTDOTS7WDUXB34VQXH5JCCFFJBA6FKQ2CGYIJFQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: <nil> Jul 28 00:46:47.718547677 (85h10m57.441110717s) [noise=true , address=104.36.82.50:4021 , piece_id=ZGODXLN7KFRHTZO7BUUR5PAPMX7ATAR7OFY5NFE4GAC446X2NZ5Q, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1nz7Gxp4YTWuB1JDnjE5Q9BfVUUvyZ4SMBbZrxVQHrSUvcJbAQ, address:104.36.82.50:4021): protocol: rpc: dial tcp 104.36.82.50:4021: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718568196 (85h10m57.441131226s) [noise=true , address=104.36.82.12:4166 , piece_id=FX4A3FB4T7NZAYL2XJ4KES4MJI23VAGQGZ7CQYJIADRN5OMF7IKQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718600618 (85h10m57.441163658s) [noise=true , address=104.36.82.13:4194 , piece_id=O54LDQ5TUJ7G7NC7ZPXVFLS4RNHQX5SLRSL3FHNODD5UGOQA7WSQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1u6tGwzk56SwgUqzpyRFLNTTysUtk6C2Ls77UkjFryeuF9D8pV, address:104.36.82.13:4194): protocol: rpc: dial tcp 104.36.82.13:4194: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718619914 (85h10m57.441182954s) [noise=true , address=104.36.82.42:4821 , piece_id=PGVY7G42RSSXOMLBYF65H6THBBAEPG74WR7JACAWNGAHAG3OQ7NQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718639292 (85h10m57.441202332s) [noise=true , address=104.36.82.11:4145 , piece_id=DS7DX6FKGY4YUCHSJAYCPN5WTUPX3CBK4X3TUSOMDXWVTWRH7GEA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718659260 (85h10m57.4412223s) [noise=true , address=104.36.82.49:4992 , piece_id=VOHZXDUQJWHPEVIET3LQUMYQIFWQUP75UDUA4AVCF7J4UO3VK5JA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv, address:104.36.82.49:4992): protocol: write tcp 188.166.225.170:34630->104.36.82.49:4992: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718677534 (85h10m57.441240574s) [noise=true , address=104.36.82.47:4535 , piece_id=TAWFMYURBQRXOUHICDDDFYKD46EJZVOD64QPLPVYYIG6WI6BRL6A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718693505 (85h10m57.441256545s) [noise=true , address=104.36.82.14:4242 , piece_id=DPAD7IVXUNT3LRHWKNOOIA72QRRWBLE5FW7D4A2HOGZ6HUG6OKQA, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: rpc: dial tcp 104.36.82.14:4242: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718712502 (85h10m57.441275542s) [noise=true , address=104.36.82.43:4851 , piece_id=54JSAT5N2PXHEAJDVMBCEN2CDJTTJSDW32QYIBN7QIHGII2ZZF7A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1nvd5hX9Wuih1hAtNzK4XAHcaSoFARdfRJijxByE4TW5HSpy4b, address:104.36.82.43:4851): protocol: rpc: dial tcp 104.36.82.43:4851: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718726338 (85h10m57.441289378s) [noise=true , address=104.36.82.54:4179 , piece_id=FPUMX7NO22OSSSXCYBFRNFKU67HP4IOG6ENHY45F3EGHAKPEXBBQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP, address:104.36.82.54:4179): protocol: rpc: dial tcp 104.36.82.54:4179: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718747168 (85h10m57.441310208s) [noise=true , address=104.36.82.53:4153 , piece_id=3ATH3AA56U4366MCMZADEAZ63HGCDFVFQPIWLQI735EPKJD4A54Q, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1fPp4DvLe18dZWPmMPFiC8o9jL4uiEFNmPAHHmFEH74ZXNndw3, address:104.36.82.53:4153): protocol: rpc: dial tcp 104.36.82.53:4153: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718761696 (85h10m57.441324736s) [noise=true , address=104.36.82.56:4235 , piece_id=VYXBYYD4UEP65672CE2HZ3LLJ6SKBXUYXDQGARIVTQFNU4Y4W2FQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: rpc: dial tcp 104.36.82.56:4235: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718780612 (85h10m57.441343652s) [noise=true , address=104.36.82.55:4201 , piece_id=Z6DY5KF6M2MRW24ORMVFPBWK63OKE2NULGZWNZA6STKRQ3KZ4VOQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12bPDvhDfu8Z58LxZcicg1vAiiu1sZYGpwQnvuhk4t5m6yR87Fi, address:104.36.82.55:4201): protocol: rpc: dial tcp 104.36.82.55:4201: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718793828 (85h10m57.441356858s) [noise=true , address=104.36.82.21:4489 , piece_id=D6NJIWRDB7TDOYNUGBB3KNBQO4CFAVTRFDVQ74JEJBTNZNVFKL5A, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:12fao48Sd4XTApMqhudk7rT4pk6FssZRoqNdoUnrkL2b8cyExc5, address:104.36.82.21:4489): protocol: rpc: dial tcp 104.36.82.21:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718827983 (85h10m57.441391023s) [noise=true , address=104.36.82.40:4971 , piece_id=DYFZ7K7DEX5OYIR7A2HWBTXDCPI6Y5BBXEWBC6S5OB3WHRRGXZJQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH, address:104.36.82.40:4971): protocol: write tcp 188.166.225.170:41456->104.36.82.40:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718855726 (85h10m57.441418766s) [noise=true , address=104.36.82.41:4773 , piece_id=PTXD3EUPC7R2M4J3V47B6W4TXEV6VSB66HU5XFZKJDV5E2DFXURQ, seg_pos={0 0}, upload=81725]: Done uploading piece. err: ecclient: upload failed (node:17zDVXSEZfrKCemdwwe5b7STPQ8PiVe4twkXAWWczQNVz4ExJV, address:104.36.82.41:4773): protocol: write tcp 188.166.225.170:56004->104.36.82.41:4773: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:46:47.718876305 (85h10m57.441439356s) [seg_pos={0 0}, upload=81725]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:46:47.949071585 (85h10m57.671634625s) [upload=81725]: Done uploading segments... Jul 28 00:46:47.949100551 (85h10m57.671663591s) [upload=81725]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232805453s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.......X.XX....X....X.......X.....X.........] duration: 5.813182914s FAIL (2025-07-28 00:46:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.813211877s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X.X..........X.....X.X...........X............] duration: 5.848012612s FAIL (2025-07-28 00:46:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.848039112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X.......................X.......................] duration: 2.114996714s FAIL (2025-07-28 00:46:15 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 2.115030269s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [....................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.228107546s FAIL (2025-07-28 00:45:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== . Jul 28 00:44:48.305181443 (85h8m58.027744494s) [noise=true , address=104.36.82.13:4212 , piece_id=GVQVLQSBNNDX3FB6WI2THSGRNH64X5K5VQ4GY7NN3ETGUK3QEUDA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305316652 (85h8m58.027879692s) [noise=true , address=104.36.82.12:4155 , piece_id=CHREES42XYFSM2VQ7ZI33BSRLFZDTNLLPQMD3RPBZG3MURQHLCNQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305476779 (85h8m58.028039819s) [noise=true , address=104.36.82.14:4250 , piece_id=PHVCH3PG62VUCPB7A5RMWF2WQCLG7M4DDSXPNXHNWP5LSBOIDXGQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305518338 (85h8m58.028081368s) [noise=true , address=104.36.82.5:4860 , piece_id=WTSQ3IMM7CQF6MAPOEXBBRMI6PASRV6CSO6P2RCKP5275AJHBCEA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305713992 (85h8m58.028277032s) [noise=true , address=104.36.82.20:4448 , piece_id=QMSM7SN7FFOMBEGFIKDNO2Q2GADVZMRSUCIPQBFEDIOB3ND7EE4Q, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305727158 (85h8m58.028290198s) [noise=true , address=104.36.82.52:4089 , piece_id=4DSMBKFZRAVMZRB7SMPPFSDFFAAHLKQVCRTKFKQ47HRR4HXG7GNQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305958430 (85h8m58.02852147s) [noise=true , address=104.36.82.56:4248 , piece_id=GYM7GTJSISHYEQKMLOUMV6NQLCJH6DY2YJNCHVEAILNACIIG5EBQ, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:48.305983728 (85h8m58.028546768s) [noise=true , address=104.36.82.47:4538 , piece_id=ZCEFGZEY36X4YO4PC2EQAPZKFKBSHMQAS3U5IIZ5UHSDFVS7XKZA, seg_pos={0 0}, upload=81711]: Uploading piece... Jul 28 00:44:49.855716732 (85h8m59.578279772s) [upload=81711]: Next returned nil segment Jul 28 00:44:49.855731269 (85h8m59.578294319s) [upload=81711]: Waiting for error group managing segments... Jul 28 00:44:52.198343156 (85h9m1.920906196s) [noise=true , address=79.127.201.215:4349 , piece_id=YOG7A3B3JX5GYUZF4BW6DQ67BPQU76LCBIO5W6EVCORBKPYWAWMQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: <nil> Jul 28 00:45:47.218477590 (85h9m56.94104063s) [noise=true , address=104.36.82.42:4806 , piece_id=XITTX6NB2KEEBS7IQGUPJFUL2CHUUWXSAQQN37EAFMY4QF555UNA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: rpc: dial tcp 104.36.82.42:4806: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218534218 (85h9m56.941097258s) [noise=true , address=104.36.82.55:4199 , piece_id=CAKWJVRE6AGFHWZHHZ2RM37D7WC4GAFL7U4TJQAG5UUNCSOP4BUQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218609452 (85h9m56.941172492s) [noise=true , address=104.36.82.51:4068 , piece_id=CLDQKGGKXGZPYR6HU27W4SF44VSGPFZYQGVXQFOSTCOR3W5QPUDA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12Lh9dYpHCxbs7gpZ4Xt7os5Gm5Nco6jrNW1V1FHddVUEMkFfPY, address:104.36.82.51:4068): protocol: write tcp 188.166.225.170:36946->104.36.82.51:4068: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218647915 (85h9m56.941210945s) [noise=true , address=104.36.82.14:4250 , piece_id=PHVCH3PG62VUCPB7A5RMWF2WQCLG7M4DDSXPNXHNWP5LSBOIDXGQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:104.36.82.14:4250): protocol: rpc: dial tcp 104.36.82.14:4250: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218682642 (85h9m56.941245682s) [noise=true , address=104.36.82.20:4448 , piece_id=QMSM7SN7FFOMBEGFIKDNO2Q2GADVZMRSUCIPQBFEDIOB3ND7EE4Q, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218709473 (85h9m56.941272513s) [noise=true , address=104.36.82.13:4212 , piece_id=GVQVLQSBNNDX3FB6WI2THSGRNH64X5K5VQ4GY7NN3ETGUK3QEUDA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1itAUtWeFERY4AKhPxE2CRwymp8qNC6wu6uhPt2n6aK58LRu24, address:104.36.82.13:4212): protocol: rpc: dial tcp 104.36.82.13:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218825335 (85h9m56.941388375s) [noise=true , address=104.36.82.5:4860 , piece_id=WTSQ3IMM7CQF6MAPOEXBBRMI6PASRV6CSO6P2RCKP5275AJHBCEA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1Qr9fghoRVbAvNm9AMEHhhtc7JqHWBuZerBttg3KW3J3GcDjrE, address:104.36.82.5:4860): protocol: rpc: dial tcp 104.36.82.5:4860: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218876113 (85h9m56.941439153s) [noise=true , address=104.36.82.56:4248 , piece_id=GYM7GTJSISHYEQKMLOUMV6NQLCJH6DY2YJNCHVEAILNACIIG5EBQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218915518 (85h9m56.941478558s) [noise=true , address=104.36.82.10:4087 , piece_id=MMTYSPZIK6X3RD3K5AWBYGDX4PMS52B3YJID5XWHR2SP4ELQJBFQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA, address:104.36.82.10:4087): protocol: rpc: dial tcp 104.36.82.10:4087: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218952889 (85h9m56.941515929s) [noise=true , address=104.36.82.7:4944 , piece_id=2IE32AGQZCWWVJEB6ZT2COEBWEIVANKMGR2VXWZTH3KLNNMRDA7A, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:15mgZMkU28J1JwrRd4XbvZFyjp8G7CTBkJCxbzUYqKeSeqCZ9F, address:104.36.82.7:4944): protocol: rpc: dial tcp 104.36.82.7:4944: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218991634 (85h9m56.941554674s) [noise=true , address=104.36.82.38:4715 , piece_id=GL3QDV5KGRYKG4R3AUFKQHF5XB275SOPWRWGQ2XZFPJSVECNBW4A, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1dtsPeGJbDSsaR1rPCFE42aCvH8o6AsnRMSK8Dp2RmcpFxo3CR, address:104.36.82.38:4715): protocol: rpc: dial tcp 104.36.82.38:4715: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.218998336 (85h9m56.941561376s) [noise=true , address=104.36.82.15:4276 , piece_id=U4X5E3TZCNQCYNIQMXZVDIW7ZXOHP26FDPY263CKR5PM6FC7RCJQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1cdmfEY6jvjGkvS3P85H3rZKPfwspQeawUtJnb5QfwLPReHu23, address:104.36.82.15:4276): protocol: rpc: dial tcp 104.36.82.15:4276: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219029656 (85h9m56.941592696s) [noise=true , address=104.36.82.41:4785 , piece_id=QPJATXNQX3JWRVGPOJA66AT6PWDEQIWRVU2GRIBLTLTQJEMMEVHQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1gYH83ECV8wai3CZyBSU8NNJM1UsysLYHbQDwBVKZRyjsdnUwB, address:104.36.82.41:4785): protocol: rpc: dial tcp 104.36.82.41:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219045166 (85h9m56.941608206s) [noise=true , address=104.36.82.12:4155 , piece_id=CHREES42XYFSM2VQ7ZI33BSRLFZDTNLLPQMD3RPBZG3MURQHLCNQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:1yiUBiCEpAs5yDmgVfFGuxgW4R53jzMJugo8Piff3p79EEZdWr, address:104.36.82.12:4155): protocol: rpc: dial tcp 104.36.82.12:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219065614 (85h9m56.941628654s) [noise=true , address=104.36.82.2:4733 , piece_id=FYJ3EGYVRLDT4D5LZBDFSDTAM2PTNMTZ5DP7MRNGYEJWXRLTS3TA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12T7L64rXL1enKykmSocFM5ZfVeVkKJWgrHVXWbkxayuuW2FXce, address:104.36.82.2:4733): protocol: rpc: dial tcp 104.36.82.2:4733: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219083850 (85h9m56.94164689s) [noise=true , address=104.36.82.52:4089 , piece_id=4DSMBKFZRAVMZRB7SMPPFSDFFAAHLKQVCRTKFKQ47HRR4HXG7GNQ, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: rpc: dial tcp 104.36.82.52:4089: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219114628 (85h9m56.941677668s) [noise=true , address=104.36.82.47:4538 , piece_id=ZCEFGZEY36X4YO4PC2EQAPZKFKBSHMQAS3U5IIZ5UHSDFVS7XKZA, seg_pos={0 0}, upload=81711]: Done uploading piece. err: ecclient: upload failed (node:12P3N79R3vpvkpsJ4fEd6NcJtmMpszRW5KHgfYJvkHQzvSTf3i5, address:104.36.82.47:4538): protocol: write tcp 188.166.225.170:58758->104.36.82.47:4538: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:45:47.219153083 (85h9m56.941716123s) [seg_pos={0 0}, upload=81711]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:45:47.445254399 (85h9m57.167817439s) [upload=81711]: Done uploading segments... Jul 28 00:45:47.445282573 (85h9m57.167845603s) [upload=81711]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.228119239s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X.X..........X.....X.X...........X...........] duration: 5.80951247s FAIL (2025-07-28 00:45:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.809540061s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.......X.XX....X....X.......X.....X........] duration: 5.749221185s FAIL (2025-07-28 00:45:42 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.749243368s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X............X.......................X......................] duration: 1.802221244s FAIL (2025-07-28 00:45:26 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.802248136s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.......X.XX....X....X.......X.....X.......] duration: 5.784039362s FAIL (2025-07-28 00:44:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.784061764s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.....................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.236142373s FAIL (2025-07-28 00:44:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== d=IV3CSZAQDQTO62GPE66QER5I3PBHNQOFDDLKTLR2NNGTGJOB6B2Q, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.824773080 (85h7m57.54733611s) [noise=true , address=79.127.201.215:4349 , piece_id=ELFZHCVAISQ322A2JAZFT44BG24NSZMI7GYMKSBVC6SHEQ7XCDHA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825005044 (85h7m57.547568084s) [noise=true , address=104.36.82.10:4080 , piece_id=BCT46Q4SKOUOT6KOHP6CS7CWPTEZLYQUGG7CNP64HP3JC7OBGBAQ, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825188916 (85h7m57.547751976s) [noise=true , address=104.36.82.46:4509 , piece_id=3R6DJAYUAYEG37MEKQ4AHMPGL3E6J3NETCPPJXZIY6UZ46CSEUSA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825431049 (85h7m57.547994089s) [noise=true , address=104.36.82.14:4226 , piece_id=FPNBHXD7FC6MBPBT6HRSOLUST4VAHL457QYW4H3IN6WXTQX3LMTA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825655098 (85h7m57.548218128s) [noise=true , address=104.36.82.56:4242 , piece_id=VOILM4KWS4P7ALU243GOMGE64WSZC6UVPJMFSTMEG6XOMG2ZEZRA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:47.825669785 (85h7m57.548232815s) [noise=true , address=104.36.82.11:4133 , piece_id=55UVZDKUXL6D7FPVNN6WECBREEOTXE23UPQQBWQZIKENCURWKHHA, seg_pos={0 0}, upload=81695]: Uploading piece... Jul 28 00:43:48.919517680 (85h7m58.64208072s) [upload=81695]: Next returned nil segment Jul 28 00:43:48.919535524 (85h7m58.642098564s) [upload=81695]: Waiting for error group managing segments... Jul 28 00:43:49.901029062 (85h7m59.623592092s) [noise=true , address=79.127.205.245:4247 , piece_id=5R6RXFCNCZ7HVKKAWY7XX3QL7NWTLFCB2P5BE3U25VKRD2V4A7FA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: <nil> Jul 28 00:43:52.019794638 (85h8m1.742357688s) [noise=true , address=79.127.201.215:4349 , piece_id=ELFZHCVAISQ322A2JAZFT44BG24NSZMI7GYMKSBVC6SHEQ7XCDHA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: <nil> Jul 28 00:44:46.723687223 (85h8m56.446250263s) [noise=true , address=104.36.82.41:4782 , piece_id=5BJF4XKOBT27XZEXXO235UCGI2HFJ5H7OYMW3JTBMYY3IQH4NVTQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1vst2WW2D3NGbxPDDkjFQPxuwX3joAY77EdWCGXxU3H1NVGgcr, address:104.36.82.41:4782): protocol: write tcp 188.166.225.170:46992->104.36.82.41:4782: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723735055 (85h8m56.446298095s) [noise=true , address=104.36.82.20:4459 , piece_id=EVENFGHQIU4HMOGOXG6XN2YO2QKC2Q7ZDKPTZDHDIFDPGNRLXWYQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1hpq5G467oUtEaE6Cj842pHCCZ1V7EraP5EDdGdWMxz4M3Q3pA, address:104.36.82.20:4459): protocol: rpc: dial tcp 104.36.82.20:4459: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723755433 (85h8m56.446318473s) [noise=true , address=104.36.82.54:4165 , piece_id=HGG4JDZZ4V7BB6Y7FQNKJDFO2V3ZBLDXW6C3GRBJAEQZD47C6P7Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12wGUMXVEubEtki4qGnjGEsbYFB1iUauZcu3oDRKrCQsJMV3bV2, address:104.36.82.54:4165): protocol: write tcp 188.166.225.170:33922->104.36.82.54:4165: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723829265 (85h8m56.446392305s) [noise=true , address=104.36.82.6:4883 , piece_id=2L5DBUVO645XSL3LB2LPFK72LLALLUBNUBGN242DA6FST7BYH42Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12o2mgPVCn34Zny7hdDtGfQQwxXWKaEfKRXGrtkuat6mMstZiZ1, address:104.36.82.6:4883): protocol: write tcp 188.166.225.170:54914->104.36.82.6:4883: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723863691 (85h8m56.446426721s) [noise=true , address=104.36.82.21:4491 , piece_id=IV3CSZAQDQTO62GPE66QER5I3PBHNQOFDDLKTLR2NNGTGJOB6B2Q, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12Qxcfg1HCcyaydtJb4i3CtKaweXzyRZxkght7xRePgMzXX1QF6, address:104.36.82.21:4491): protocol: write tcp 188.166.225.170:46800->104.36.82.21:4491: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723931852 (85h8m56.446494892s) [noise=true , address=104.36.82.10:4080 , piece_id=BCT46Q4SKOUOT6KOHP6CS7CWPTEZLYQUGG7CNP64HP3JC7OBGBAQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12KK51wudKiZb39Ku7wZhgBYEYmgsZTHfjiNNLmpBU9uuNFapt3, address:104.36.82.10:4080): protocol: write tcp 188.166.225.170:37804->104.36.82.10:4080: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.723970174 (85h8m56.446533215s) [noise=true , address=104.36.82.46:4509 , piece_id=3R6DJAYUAYEG37MEKQ4AHMPGL3E6J3NETCPPJXZIY6UZ46CSEUSA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: write tcp 188.166.225.170:44888->104.36.82.46:4509: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724039918 (85h8m56.446603019s) [noise=true , address=104.36.82.56:4242 , piece_id=VOILM4KWS4P7ALU243GOMGE64WSZC6UVPJMFSTMEG6XOMG2ZEZRA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1Q615w5XWJT1ownaJL1JFqyqntzMj6fgNieAfR6z421YCHuR5r, address:104.36.82.56:4242): protocol: write tcp 188.166.225.170:53882->104.36.82.56:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724073713 (85h8m56.446636753s) [noise=true , address=104.36.82.55:4227 , piece_id=V3RLTAWSPAFHFWC7UUV7BNCBBQRCUXNM2LTMZUECG67H7ZWII5XQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:19qGGwWT9xUYsJyRnFHyHKPFr6g4AsckxRNVTg2shfVhoQtcCz, address:104.36.82.55:4227): protocol: write tcp 188.166.225.170:41706->104.36.82.55:4227: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724104081 (85h8m56.446667111s) [noise=true , address=104.36.82.13:4215 , piece_id=HUKA5NZGDYCHGYQ2INUAHRB3GSHOH34KNQQ7LUFVK4OGSICLGO3A, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1vV1hGhZqeEoh5ei7yRu49oyAvTZ2T7S1x3ooufJPtFH2TGBkQ, address:104.36.82.13:4215): protocol: write tcp 188.166.225.170:50148->104.36.82.13:4215: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724171099 (85h8m56.446734139s) [noise=true , address=104.36.82.12:4152 , piece_id=FYDETR5M3DV52EMJS5KI37W2KR7TO2Y7SWEIBKRETT2USGQAODOA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 188.166.225.170:41576->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724201607 (85h8m56.446764637s) [noise=true , address=104.36.82.14:4226 , piece_id=FPNBHXD7FC6MBPBT6HRSOLUST4VAHL457QYW4H3IN6WXTQX3LMTA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:126xWe3ELGmYQfSUk3tYtGTSa6LnGfDNWcGT164pR6tHEtFPAFt, address:104.36.82.14:4226): protocol: write tcp 188.166.225.170:47274->104.36.82.14:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724265179 (85h8m56.446828219s) [noise=true , address=104.36.82.11:4133 , piece_id=55UVZDKUXL6D7FPVNN6WECBREEOTXE23UPQQBWQZIKENCURWKHHA, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:12G5pngZWYQ2abKHEpM7Pa6YeCQ1LotX4qswbqHUwAy9H7a9Ym2, address:104.36.82.11:4133): protocol: write tcp 188.166.225.170:43966->104.36.82.11:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724304764 (85h8m56.446867805s) [noise=true , address=104.36.82.53:4152 , piece_id=NNU6SV4MWUH62AA4WYJGHUPTTQ3HHJYTBNJ45O4YGOJAZRAXYVCQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 188.166.225.170:50952->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724341045 (85h8m56.446904075s) [noise=true , address=104.36.82.41:4781 , piece_id=CPZA7NVSTTNPBLUDLZPANKINOWBIBWRR2PQI7B3ZVLCHBUQNAMCQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1QhCkTzSogXFKLmFV4wQUcALhvm9YhCSQudLxSrvQWmepheiFi, address:104.36.82.41:4781): protocol: write tcp 188.166.225.170:42670->104.36.82.41:4781: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724353057 (85h8m56.446916097s) [noise=true , address=104.36.82.42:4806 , piece_id=UR4V3CQ5LKZHDUPST6GOXMPDMJV4R65KSMPQQ42MJ3Q744CEM7ZQ, seg_pos={0 0}, upload=81695]: Done uploading piece. err: ecclient: upload failed (node:1tWnxuTSo4yHAJUzvVFTmwPSNb8cBJJJ17BnKd5Z3HMYf66vSC, address:104.36.82.42:4806): protocol: write tcp 188.166.225.170:38162->104.36.82.42:4806: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:44:46.724376472 (85h8m56.446939512s) [seg_pos={0 0}, upload=81695]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:44:46.958002645 (85h8m56.680565685s) [upload=81695]: Done uploading segments... Jul 28 00:44:46.958029457 (85h8m56.680592497s) [upload=81695]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.236158894s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X.X..........X.....X.X...........X..........] duration: 5.813978415s FAIL (2025-07-28 00:44:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.81400848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X............X.......................X.....................] duration: 1.994745419s FAIL (2025-07-28 00:44:34 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.994773633s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X.X..........X.....X.X...........X.........] duration: 5.87200132s FAIL (2025-07-28 00:44:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872032671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.......X.XX....X....X.......X.....X......] duration: 5.746031143s FAIL (2025-07-28 00:44:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.746058946s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.......X.XX....X....X.......X.....X.....] duration: 5.841496136s FAIL (2025-07-28 00:43:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.841518157s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X.X..........X.....X.X...........X........] duration: 5.712918278s FAIL (2025-07-28 00:43:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712944629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [......................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232668719s FAIL (2025-07-28 00:43:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 43 (85h6m57.064271683s) [noise=true , address=104.36.82.12:4158 , piece_id=XQZOWWS7LKNWRV4QVB5COFXHOSHOOMHIHHDYZFZPBDWN5PQNG7KA, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.341833061 (85h6m57.064396101s) [noise=true , address=104.36.82.4:4809 , piece_id=2SD7SIVSXJPQEPZXEK7354H3YFTRNA3T63NY7UMHTAZGPWMGXMJQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342059063 (85h6m57.064622113s) [noise=true , address=104.36.82.15:4260 , piece_id=MOUIJB5XJUIOWPHFS7OKBMT6W42BQEWMXANFUCKFMO4RBVPJLMJQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342084622 (85h6m57.064647662s) [noise=true , address=104.36.82.22:4509 , piece_id=6ADNMOVSC4NVUYYSNVR5XLRPV5POGVFFK7ZWKNWD2EGUVKZV3RMQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342403092 (85h6m57.064966132s) [noise=true , address=104.36.82.5:4842 , piece_id=JIRKR4WXYSVKRBADPJ7V6WLBNPTX4R55KQRN32MZUMV2T63WX5PA, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342409093 (85h6m57.064972133s) [noise=true , address=104.36.82.14:4240 , piece_id=C27DS6GK5LBLW6DNQ6OQMF4PVQZ753BFZQMK5K3UTDDZRNNTWR7Q, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342714468 (85h6m57.065277498s) [noise=true , address=104.36.82.46:4482 , piece_id=CI7ZROXKTZKIFFCGNMGSO35JKKXZYLPZKDGPWYDCDPHKLZTCVERQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342770655 (85h6m57.065333705s) [noise=true , address=79.127.201.215:4352 , piece_id=IKKYD4KSSEOCIEAXGMMQLUUG3CFH73E7E3SI5R7TI6NOBZXRCMKQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342922767 (85h6m57.065485807s) [noise=true , address=104.36.82.54:4158 , piece_id=LD5WKHMVJVB7MPZ5LA64SNHJI32P3TZL4HK3NIUBPX3RLD3TLVDQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:48.455780921 (85h6m58.178343961s) [upload=81675]: Next returned nil segment Jul 28 00:42:48.455802422 (85h6m58.178365472s) [upload=81675]: Waiting for error group managing segments... Jul 28 00:42:48.784431513 (85h6m58.506994564s) [noise=true , address=79.127.205.235:4688 , piece_id=TS2LM64VH7M7MRJJ2TND6WSBRD2UFIO5RQWHXL3BYPNTM4DQXIQQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: <nil> Jul 28 00:42:49.343244142 (85h6m59.065807213s) [noise=true , address=79.127.201.215:4352 , piece_id=IKKYD4KSSEOCIEAXGMMQLUUG3CFH73E7E3SI5R7TI6NOBZXRCMKQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: <nil> Jul 28 00:43:46.260330313 (85h7m55.982893353s) [noise=true , address=104.36.82.53:4148 , piece_id=VWEU2KT72MZRWNM2HHKUEX4ICI5OSZ4MHTDFHPDVTW3C4KWSX4KQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260423952 (85h7m55.982986992s) [noise=true , address=104.36.82.14:4240 , piece_id=C27DS6GK5LBLW6DNQ6OQMF4PVQZ753BFZQMK5K3UTDDZRNNTWR7Q, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260482554 (85h7m55.983045594s) [noise=true , address=104.36.82.46:4482 , piece_id=CI7ZROXKTZKIFFCGNMGSO35JKKXZYLPZKDGPWYDCDPHKLZTCVERQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260528673 (85h7m55.983091703s) [noise=true , address=104.36.82.3:4788 , piece_id=MZSH6QGBY67U6AX436UQQINE7654T2PALNLVENX63ISINDHH5EMQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: rpc: dial tcp 104.36.82.3:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260550083 (85h7m55.983113123s) [noise=true , address=104.36.82.54:4158 , piece_id=LD5WKHMVJVB7MPZ5LA64SNHJI32P3TZL4HK3NIUBPX3RLD3TLVDQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: rpc: dial tcp 104.36.82.54:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260562578 (85h7m55.983125618s) [noise=true , address=104.36.82.15:4260 , piece_id=MOUIJB5XJUIOWPHFS7OKBMT6W42BQEWMXANFUCKFMO4RBVPJLMJQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1aLsgrV3k6vUSavgPkPGA2FF9HR3ryN9bnHutruFJHW1GYixeL, address:104.36.82.15:4260): protocol: rpc: dial tcp 104.36.82.15:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260594498 (85h7m55.983157538s) [noise=true , address=104.36.82.44:4891 , piece_id=7RKDP66EPDDTYVKLRDWVYM3JIUR3APDZGDHWXDDE2257UPBCNETA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260599959 (85h7m55.983162989s) [noise=true , address=104.36.82.22:4509 , piece_id=6ADNMOVSC4NVUYYSNVR5XLRPV5POGVFFK7ZWKNWD2EGUVKZV3RMQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:104.36.82.22:4509): protocol: rpc: dial tcp 104.36.82.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260655365 (85h7m55.983218405s) [noise=true , address=104.36.82.13:4204 , piece_id=6JCSUI2O26UGR45QDIAQGKBW6W4I4GL5JSANEKA3WX5WBJ2PRVIA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260672538 (85h7m55.983235578s) [noise=true , address=104.36.82.56:4253 , piece_id=V2U5R3XL4DVCM36KXZLIZ6AAWOVKHTOZQ36KLPSWYOHW7GF2PDGA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260705721 (85h7m55.983268761s) [noise=true , address=104.36.82.55:4210 , piece_id=LXLEOZRR63IOBKCNUGKCL5HAQFMFP5WLYDY37JHTL5MZ5MEX3BTA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260710109 (85h7m55.983273149s) [noise=true , address=104.36.82.12:4158 , piece_id=XQZOWWS7LKNWRV4QVB5COFXHOSHOOMHIHHDYZFZPBDWN5PQNG7KA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260745348 (85h7m55.983308388s) [noise=true , address=104.36.82.10:4089 , piece_id=ED5J6MS3NPOT7LLN75PHJYT5NTWH43UUUFFALC3NDHSR4WTYD3CA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:47062->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260751860 (85h7m55.9833149s) [noise=true , address=104.36.82.5:4842 , piece_id=JIRKR4WXYSVKRBADPJ7V6WLBNPTX4R55KQRN32MZUMV2T63WX5PA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: write tcp 188.166.225.170:43612->104.36.82.5:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260760095 (85h7m55.983323135s) [noise=true , address=104.36.82.4:4809 , piece_id=2SD7SIVSXJPQEPZXEK7354H3YFTRNA3T63NY7UMHTAZGPWMGXMJQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9, address:104.36.82.4:4809): protocol: rpc: dial tcp 104.36.82.4:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260795423 (85h7m55.983358463s) [noise=true , address=104.36.82.4:4828 , piece_id=ISMIYHIB2GV5LUVOK5FV2NIQORBYIXOSEEPRP5EKG7W27LUFHPOA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:46250->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260812075 (85h7m55.983375115s) [seg_pos={0 0}, upload=81675]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:43:46.490922522 (85h7m56.213485552s) [upload=81675]: Done uploading segments... Jul 28 00:43:46.490950155 (85h7m56.213513195s) [upload=81675]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232693575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X............X.......................X....................] duration: 1.868696121s FAIL (2025-07-28 00:43:43 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.868719375s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X.......X.XX....X....X.......X.....X....] duration: 5.745423669s FAIL (2025-07-28 00:43:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.745453705s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X.X..........X.....X.X...........X.......] duration: 6.044680756s FAIL (2025-07-28 00:43:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.044710814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X............X.......................X...................] duration: 1.897747454s FAIL (2025-07-28 00:43:18 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.897778634s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X.......X.XX....X....X.......X.....X...] duration: 5.736130227s FAIL (2025-07-28 00:43:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.736156599s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X............X.......................X..................] duration: 1.893015371s FAIL (2025-07-28 00:43:02 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.893045677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X.X..........X.....X.X...........X......] duration: 5.767336717s FAIL (2025-07-28 00:42:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.76736408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.......................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239594771s FAIL (2025-07-28 00:42:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== ce... Jul 28 00:41:39.003381710 (85h5m48.72594475s) [noise=true , address=79.127.201.215:4355 , piece_id=J4K4L3ZOPZR6YGS6E3YHPHP5QCFQU632SITGREMOGYWPSO3GD42Q, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003526477 (85h5m48.726089518s) [noise=true , address=104.36.82.14:4224 , piece_id=3CLJUK32K65AU5CTJBUORQKBDF7MJVPVF5S6CTVPWGASWZ7ZGWLA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003641417 (85h5m48.726204457s) [noise=true , address=104.36.82.11:4148 , piece_id=BOTXTYSNMNTURBEGQWMVIU4T7BZZVBSRL5FMUJIGBC6TJMCQUBBA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003743792 (85h5m48.726306832s) [noise=true , address=104.36.82.13:4191 , piece_id=V3EJ56GLXLTGUOJZH7GEPDH4HW4IPWTCWGUVYJXYZ2DXRUH6QKBA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003912306 (85h5m48.726475356s) [noise=true , address=79.127.220.101:4045 , piece_id=XDCT3BR5WLV2ZY5PN74AOZQFXLK4ZJSGMIW6X37OBMECTTYKWIWQ, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.004316759 (85h5m48.726879799s) [noise=true , address=104.36.82.52:4094 , piece_id=P3K6JDUVTQTFRLGVETTRKXYF7DHWAGFGUBB7W7FY3IOKDWY5N62A, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.004597225 (85h5m48.727160265s) [noise=true , address=104.36.82.41:4788 , piece_id=CVBDBO3ATBQQNCZ2GXTTBETJTCJHYN5FY72VYHMD7CPSTV5I2X5A, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.005053248 (85h5m48.727616288s) [noise=true , address=104.36.82.42:4816 , piece_id=WQ3EASTEHG2BSNCWE62I4H3L5OZX35O37Q3IONF3CSYPAGQHN5QA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:40.406283592 (85h5m50.128846632s) [upload=81659]: Next returned nil segment Jul 28 00:41:40.406300505 (85h5m50.128863545s) [upload=81659]: Waiting for error group managing segments... Jul 28 00:41:40.828734633 (85h5m50.551297673s) [noise=true , address=79.127.220.101:4045 , piece_id=XDCT3BR5WLV2ZY5PN74AOZQFXLK4ZJSGMIW6X37OBMECTTYKWIWQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:41:42.498903904 (85h5m52.221466944s) [noise=true , address=79.127.201.215:4356 , piece_id=HPHM7PLAXQLE627NSVB54ZYJFS2D3AIHE56G4AOOQEISUGXND5YA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:41:51.457045956 (85h6m1.179609006s) [noise=true , address=79.127.201.215:4355 , piece_id=J4K4L3ZOPZR6YGS6E3YHPHP5QCFQU632SITGREMOGYWPSO3GD42Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:42:37.858869818 (85h6m47.581432858s) [noise=true , address=104.36.82.14:4224 , piece_id=3CLJUK32K65AU5CTJBUORQKBDF7MJVPVF5S6CTVPWGASWZ7ZGWLA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: rpc: dial tcp 104.36.82.14:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.858987814 (85h6m47.581550854s) [noise=true , address=104.36.82.54:4183 , piece_id=6KZME7RH5QJSS43BTZUHKSWT3GGS6IGCMTH46SJBDVEXC66WCAYA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:34178->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859169261 (85h6m47.581732311s) [noise=true , address=104.36.82.41:4788 , piece_id=CVBDBO3ATBQQNCZ2GXTTBETJTCJHYN5FY72VYHMD7CPSTV5I2X5A, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 188.166.225.170:48552->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859264493 (85h6m47.581827533s) [noise=true , address=104.36.82.50:4036 , piece_id=VBIQBIVTDWYTSTJK3LXO6357H56R3DGZLDU7TCBFBVXMOVZOLI5Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:17cebKoBWpDfVVnHqUDtFfJ71Lz8B7DqFonLumECeHxHf5o4sY, address:104.36.82.50:4036): protocol: rpc: dial tcp 104.36.82.50:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859336050 (85h6m47.58189909s) [noise=true , address=104.36.82.8:4979 , piece_id=MYUANZUUFWMPM4GBPRG3723DTSLEX2UE7GSGU7FJ5PHT7YHK4D3Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 188.166.225.170:45364->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859404742 (85h6m47.581967782s) [noise=true , address=104.36.82.13:4191 , piece_id=V3EJ56GLXLTGUOJZH7GEPDH4HW4IPWTCWGUVYJXYZ2DXRUH6QKBA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:104.36.82.13:4191): protocol: write tcp 188.166.225.170:59394->104.36.82.13:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859461381 (85h6m47.582024421s) [noise=true , address=104.36.82.52:4094 , piece_id=P3K6JDUVTQTFRLGVETTRKXYF7DHWAGFGUBB7W7FY3IOKDWY5N62A, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 188.166.225.170:46340->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859867066 (85h6m47.582430116s) [noise=true , address=104.36.82.42:4816 , piece_id=WQ3EASTEHG2BSNCWE62I4H3L5OZX35O37Q3IONF3CSYPAGQHN5QA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:104.36.82.42:4816): protocol: write tcp 188.166.225.170:53552->104.36.82.42:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859955466 (85h6m47.582518516s) [noise=true , address=104.36.82.53:4133 , piece_id=QGV4FYPXFALP4AUPUEMJIATIZ6Q2G7NWHEQU24WQYOCXFEFHHS4Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 188.166.225.170:32996->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860042943 (85h6m47.582605993s) [noise=true , address=104.36.82.55:4221 , piece_id=FLJIATJ6PKQYCLWCMQPUVX2NIN6UFOWLUYEO4MDE3D47FFIYPVWA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: write tcp 188.166.225.170:33910->104.36.82.55:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860129338 (85h6m47.582692378s) [noise=true , address=104.36.82.21:4497 , piece_id=LM5DLP2VY2GJMIBWIPPPRLMHLCAA3IB7Q4ZF273NRIA4TGTMSJAA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: write tcp 188.166.225.170:35874->104.36.82.21:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860152032 (85h6m47.582715082s) [noise=true , address=104.36.82.56:4249 , piece_id=ENQ5BFW7XGHUBVNQT7XFKG2QW2YBGP2CQLEEMPUGLJ4CXVQKK3YQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 188.166.225.170:60768->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860210534 (85h6m47.582773574s) [noise=true , address=104.36.82.9:4023 , piece_id=6EXTQEATJJPFCFK35VUT25UHRYLGKYKSRMVOAXJEPIEMW3LSH3DQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:4023): protocol: write tcp 188.166.225.170:57586->104.36.82.9:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860250580 (85h6m47.58281363s) [noise=true , address=104.36.82.11:4148 , piece_id=BOTXTYSNMNTURBEGQWMVIU4T7BZZVBSRL5FMUJIGBC6TJMCQUBBA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12Rkh9uZUA5jXBVZvvt1jE7GHiEszKvR9PweKEry2AkdwoCiANs, address:104.36.82.11:4148): protocol: write tcp 188.166.225.170:43986->104.36.82.11:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860256863 (85h6m47.582819903s) [noise=true , address=104.36.82.12:4183 , piece_id=7UPVLWN5KN6VMYKRFF3S3AD6A67TGVR4BA454FY6RJQ4BBS7OWCQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: write tcp 188.166.225.170:56608->104.36.82.12:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860274676 (85h6m47.582837716s) [seg_pos={0 0}, upload=81659]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:42:38.096205640 (85h6m47.81876868s) [upload=81659]: Done uploading segments... Jul 28 00:42:38.096230938 (85h6m47.818793978s) [upload=81659]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.239607084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X.......X.XX....X....X.......X.....X..] duration: 5.779685161s FAIL (2025-07-28 00:42:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.779713937s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X.......................X.................] duration: 1.896177581s FAIL (2025-07-28 00:42:12 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.896203682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X............X.......................X................] duration: 1.902463939s FAIL (2025-07-28 00:41:39 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.902488435s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230841955s FAIL (2025-07-28 00:41:37 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== 5VBSJQIAD6KT7TA, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.483904554 (85h4m48.206467584s) [noise=true , address=104.36.82.47:4532 , piece_id=ZUXRGJEKLS5BN6N2F2VJACTGSDCKE5AGDH6M4HAMIZEFYT4A3Z5A, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484095570 (85h4m48.20665861s) [noise=true , address=104.36.82.35:4611 , piece_id=4BOT7PNPJMBFDZXOZVMEML723KJ7RZ5EYANGGD3N4L2KMRYICLLA, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484291685 (85h4m48.206854715s) [noise=true , address=104.36.82.41:4765 , piece_id=NJNR7ZP4AHPM7QHLFHV3ZMVKH5C7EH7PZ5B3D2DKSQLNSZXYUOUQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484528588 (85h4m48.207091628s) [noise=true , address=104.36.82.52:4094 , piece_id=MBEKPZYUK625WWXP2KIP3BFV6POQJDFRG4MQZWZNHYUGF4YXB5LQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484733320 (85h4m48.20729636s) [noise=true , address=104.36.82.12:4159 , piece_id=2MHN6XSDHHCCUUQQQJQ2MFO5GGWZAOVTUF3STXGG7ZAXTQSJEZWQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484932951 (85h4m48.207495991s) [noise=true , address=104.36.82.14:4244 , piece_id=BHAGCPT6S6GL6FTE2QYHFEKBC6HR5O7A42OQNPVBXYHLWJGYJABQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.485117154 (85h4m48.207680194s) [noise=true , address=104.36.82.15:4266 , piece_id=ZPNNDC26LLPZZLOKNHDNSSDX3BUUQTQOH2YBPM5LD5T5VG3CFAUQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.485299373 (85h4m48.207862413s) [noise=true , address=104.36.82.53:4154 , piece_id=NYQSCKB5E326IZKFU6ARQM4E4AJWOC34T3J3CNN2DY4QLULN74FQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:41:37.397645517 (85h5m47.120208557s) [upload=81644]: Next segment err: context deadline exceeded Jul 28 00:41:37.398329837 (85h5m47.120892867s) [noise=true , address=104.36.82.35:4611 , piece_id=4BOT7PNPJMBFDZXOZVMEML723KJ7RZ5EYANGGD3N4L2KMRYICLLA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc, address:104.36.82.35:4611): protocol: rpc: dial tcp 104.36.82.35:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398370724 (85h5m47.120933764s) [noise=true , address=104.36.82.41:4765 , piece_id=NJNR7ZP4AHPM7QHLFHV3ZMVKH5C7EH7PZ5B3D2DKSQLNSZXYUOUQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:104.36.82.41:4765): protocol: rpc: dial tcp 104.36.82.41:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398400933 (85h5m47.120963973s) [noise=true , address=104.36.82.47:4536 , piece_id=HP4UF6UP6FURKBUEDANDS7JTFU4XIRITAEUKQ6UUZ4XICOC2GEVQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 188.166.225.170:43872->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398433756 (85h5m47.120996786s) [noise=true , address=104.36.82.9:4026 , piece_id=XS3MZD67O24ELSDA7KLSZEA7ELBPU5ZU3CZV5WRURGZZ2WJMJVGQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq, address:104.36.82.9:4026): protocol: rpc: dial tcp 104.36.82.9:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398461869 (85h5m47.121024909s) [noise=true , address=104.36.82.52:4094 , piece_id=MBEKPZYUK625WWXP2KIP3BFV6POQJDFRG4MQZWZNHYUGF4YXB5LQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398495073 (85h5m47.121058113s) [noise=true , address=104.36.82.12:4159 , piece_id=2MHN6XSDHHCCUUQQQJQ2MFO5GGWZAOVTUF3STXGG7ZAXTQSJEZWQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398526703 (85h5m47.121089743s) [noise=true , address=104.36.82.14:4244 , piece_id=BHAGCPT6S6GL6FTE2QYHFEKBC6HR5O7A42OQNPVBXYHLWJGYJABQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: write tcp 188.166.225.170:60544->104.36.82.14:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398565909 (85h5m47.121128949s) [noise=true , address=104.36.82.15:4266 , piece_id=ZPNNDC26LLPZZLOKNHDNSSDX3BUUQTQOH2YBPM5LD5T5VG3CFAUQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j, address:104.36.82.15:4266): protocol: rpc: dial tcp 104.36.82.15:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398584574 (85h5m47.121147614s) [noise=true , address=104.36.82.42:4821 , piece_id=NZA2C62FVEJ7JM74GIKAQU77XIVW63NUJ26DCDCWIRLD2GSTOSVQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398630552 (85h5m47.121193592s) [noise=true , address=104.36.82.47:4532 , piece_id=ZUXRGJEKLS5BN6N2F2VJACTGSDCKE5AGDH6M4HAMIZEFYT4A3Z5A, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398659948 (85h5m47.121222988s) [noise=true , address=104.36.82.56:4259 , piece_id=PYVSMYNYJU5RTL4X3NBYRF27NRWQ3XPJL6ICU2IXCM6RO35WPDBQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: rpc: dial tcp 104.36.82.56:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398683944 (85h5m47.121246974s) [noise=true , address=104.36.82.13:4197 , piece_id=UAYGPQPDFZIYM6M4VZR24HF42CUHXNAKCHTSU6FNKCSBHEGGPTMA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: rpc: dial tcp 104.36.82.13:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398688813 (85h5m47.121251843s) [noise=true , address=104.36.82.54:4185 , piece_id=QEGMEMSFPEGRDOREOT6ZUW4LDZGT7EIUEO6YO5VBSJQIAD6KT7TA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: rpc: dial tcp 104.36.82.54:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398720634 (85h5m47.121283664s) [noise=true , address=104.36.82.10:4098 , piece_id=IDQFW3G3EAQZ52ZB77LX44TEWXJGCOHURAD3D2ESUJYRA4QMENRQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: rpc: dial tcp 104.36.82.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398732848 (85h5m47.121295878s) [noise=true , address=104.36.82.4:4830 , piece_id=P56LXY4ZIMBUGXUEGJ6DLFRS3QUTABZ4PP6W77FHPXDN24E36OXQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1p91PtS64Ue8JkPmbh2f3gig8cTQ3AMdDJPaP1249h4XmXUJrp, address:104.36.82.4:4830): protocol: rpc: dial tcp 104.36.82.4:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398739220 (85h5m47.12130226s) [noise=true , address=104.36.82.44:4870 , piece_id=2X6TTJTEUIRPUGE75YEIO66LJFV5JQ35WPLTKN6KA3NFTLQSAPRA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398744019 (85h5m47.121307059s) [noise=true , address=104.36.82.53:4154 , piece_id=NYQSCKB5E326IZKFU6ARQM4E4AJWOC34T3J3CNN2DY4QLULN74FQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398751233 (85h5m47.121314273s) [noise=true , address=104.36.82.5:4864 , piece_id=HLG6U6REL3VKPA5FQIPUI3LI5WWVXXRG6RYL5LT6XQBJMIZL36QA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: write tcp 188.166.225.170:51450->104.36.82.5:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398768416 (85h5m47.121331446s) [seg_pos={0 0}, upload=81644]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.230855491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.X..........X.....X.X...........X.....] duration: 5.819633605s FAIL (2025-07-28 00:41:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.8196572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X.......X.XX....X....X.......X.....X.] duration: 5.885188643s FAIL (2025-07-28 00:41:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.885217196s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X.......................X...............] duration: 1.850004538s FAIL (2025-07-28 00:41:12 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.850039105s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X.......X.XX....X....X.......X.....X] duration: 30.238408019s FAIL (2025-07-28 00:40:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.238430632s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [.........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232923955s FAIL (2025-07-28 00:40:37 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== ess=104.36.82.16:4313 , piece_id=EZC2I757SSU77IGFOJ62TI5OA4CPD3LZYSU5GAPQZ3KQCE6C7NOA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031296221 (85h3m47.753859261s) [noise=true , address=104.36.82.54:4169 , piece_id=MTILMOCHCF47BKDWJ3BEJSGKT3FM5AETFKO3MM5726DUI6SQANQA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031408105 (85h3m47.753971135s) [noise=true , address=104.36.82.11:4120 , piece_id=UUEIVUSY7OSDGL6V6MYA7BMMJQ6Z6DTWZNHC72M57JZLS3TDSPEA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031682440 (85h3m47.75424564s) [noise=true , address=104.36.82.46:4509 , piece_id=DD3H22UHCU2NUQB3I7GNUKSNTYK2SYVPBI7HHOHWPQVXJHVI3SGA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031692389 (85h3m47.754255429s) [noise=true , address=104.36.82.44:4886 , piece_id=UVFREUL5IVXMTEUALTEP4Z46RY6XOGGZSYX3CZ73DKRRVNP272DQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.032495757 (85h3m47.755058797s) [noise=true , address=104.36.82.4:4828 , piece_id=OB2O5EOVBUDNAS67MZS5SJHMWZRRYGJE6GMJ3BMLRKGM3MISX2LQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.032924907 (85h3m47.755487947s) [noise=true , address=104.36.82.10:4101 , piece_id=2GNAF6R4VL7IP4QZHH6QMKBLRVZ56XSZYX5ZMPYG4XH7XJK4D2AQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:40:36.944988797 (85h4m46.667551847s) [upload=81625]: Next segment err: context deadline exceeded Jul 28 00:40:36.945542306 (85h4m46.668105346s) [noise=true , address=104.36.82.56:4236 , piece_id=OHP5ABJYYV4BVTGCZVIJLYRCEZSH2ZKSFAN6IQVUJCNQRVUWRBMA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:36016->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945582442 (85h4m46.668145482s) [noise=true , address=104.36.82.55:4200 , piece_id=2J4PE7XQTCE6QPB3AC4RDBIJ2VAS6T4FOILNQVUH7D3YZHD73FWA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945610476 (85h4m46.668173516s) [noise=true , address=104.36.82.47:4535 , piece_id=HYAE6ZMAGOJTI4Q563X6HB6DIAEIRJYDQ4AKDNLWD232RSP45THQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945652727 (85h4m46.668215767s) [noise=true , address=104.36.82.14:4246 , piece_id=VNGLR3VR4LZB2V5AR2JH5BFSIN4FCXY2Y2Y5UFNL4SE5ENFHWNAA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12dqLB6LVQJaSdEfjKmu5nXqRST733hn5rTxCcMQS8YJVv8Hwdc, address:104.36.82.14:4246): protocol: write tcp 188.166.225.170:58426->104.36.82.14:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945685009 (85h4m46.668248049s) [noise=true , address=104.36.82.54:4169 , piece_id=MTILMOCHCF47BKDWJ3BEJSGKT3FM5AETFKO3MM5726DUI6SQANQA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 188.166.225.170:52136->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945720085 (85h4m46.668283125s) [noise=true , address=104.36.82.11:4120 , piece_id=UUEIVUSY7OSDGL6V6MYA7BMMJQ6Z6DTWZNHC72M57JZLS3TDSPEA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: rpc: dial tcp 104.36.82.11:4120: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945769600 (85h4m46.66833265s) [noise=true , address=104.36.82.10:4101 , piece_id=2GNAF6R4VL7IP4QZHH6QMKBLRVZ56XSZYX5ZMPYG4XH7XJK4D2AQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:45392->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945824155 (85h4m46.668387195s) [noise=true , address=104.36.82.43:4857 , piece_id=GPSOWXYUBBNEIBFDNCNC22P36RFLTNAAQG4PUJUKSEJB54XLWUXA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1DK5cJhprAFbNCSceZ527v5rMFw7e3teUTsxBXVPcjXUDiVv4P, address:104.36.82.43:4857): protocol: write tcp 188.166.225.170:38522->104.36.82.43:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946087538 (85h4m46.668650578s) [noise=true , address=104.36.82.12:4167 , piece_id=LIZXSRCIVBZWXPAKNL2LZYQUK26UTEGIQOWW3BHRRSQFE6VJ25ZQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: rpc: dial tcp 104.36.82.12:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946134198 (85h4m46.668697238s) [noise=true , address=104.36.82.47:4532 , piece_id=4N5ZM6ADCNWF3TMZ7A555WYURY4RGAKQ6OTOJTALZ2T5RQU3XG2Q, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946178353 (85h4m46.668741393s) [noise=true , address=104.36.82.53:4121 , piece_id=KA2RIAAOMBPF5N47RSL44R2L6L2PXISD7IUQUXRHLUAHHCRBEF3A, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 188.166.225.170:57456->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946216084 (85h4m46.668779124s) [noise=true , address=104.36.82.44:4886 , piece_id=UVFREUL5IVXMTEUALTEP4Z46RY6XOGGZSYX3CZ73DKRRVNP272DQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:49626->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946248606 (85h4m46.668811646s) [noise=true , address=104.36.82.47:4545 , piece_id=N767D2NVW22SIHYCCMPBIS3RG75CSUBR4ABESUXGWJPTU7PN5X2A, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 188.166.225.170:37190->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946409735 (85h4m46.668972775s) [noise=true , address=104.36.82.4:4828 , piece_id=OB2O5EOVBUDNAS67MZS5SJHMWZRRYGJE6GMJ3BMLRKGM3MISX2LQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:36886->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946456244 (85h4m46.669019284s) [noise=true , address=104.36.82.13:4218 , piece_id=MVZWT5N3CS4XCLCJCAM7BMZYFZHPQYGVNHUQU73GB6YNITTCM74Q, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: write tcp 188.166.225.170:39014->104.36.82.13:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946504437 (85h4m46.669067477s) [noise=true , address=104.36.82.5:4842 , piece_id=3IHSL6RN7IV2K4XGODMY47543RKRHNT6OC7KZIUI3GMGYU3EWKNQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: write tcp 188.166.225.170:52642->104.36.82.5:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946848965 (85h4m46.669412005s) [noise=true , address=104.36.82.16:4313 , piece_id=EZC2I757SSU77IGFOJ62TI5OA4CPD3LZYSU5GAPQZ3KQCE6C7NOA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:104.36.82.16:4313): protocol: rpc: dial tcp 104.36.82.16:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.947176843 (85h4m46.669739883s) [noise=true , address=104.36.82.46:4509 , piece_id=DD3H22UHCU2NUQB3I7GNUKSNTYK2SYVPBI7HHOHWPQVXJHVI3SGA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:40:36.947196170 (85h4m46.6697592s) [seg_pos={0 0}, upload=81625]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.232937061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X.X..........X.....X.X...........X....] duration: 5.910255022s FAIL (2025-07-28 00:40:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.910286762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X............X.......................X..............] duration: 1.810646015s FAIL (2025-07-28 00:40:32 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.810681062s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X.......X.XX....X....X.......X.....] duration: 5.885959664s FAIL (2025-07-28 00:39:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.885982738s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X.X..........X.....X.X...........X...] duration: 5.777641359s FAIL (2025-07-28 00:39:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.77766315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........X............X.......................X.............] duration: 1.836720474s FAIL (2025-07-28 00:39:40 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.836744711s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)