Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X.X..........X.....X.X...........X.........] duration: 5.87200132s FAIL (2025-07-28 00:44:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.872032671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.......X.XX....X....X.......X.....X......] duration: 5.746031143s FAIL (2025-07-28 00:44:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.746058946s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.......X.XX....X....X.......X.....X.....] duration: 5.841496136s FAIL (2025-07-28 00:43:54 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.841518157s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X.X..........X.....X.X...........X........] duration: 5.712918278s FAIL (2025-07-28 00:43:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712944629s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [......................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232668719s FAIL (2025-07-28 00:43:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== 43 (85h6m57.064271683s) [noise=true , address=104.36.82.12:4158 , piece_id=XQZOWWS7LKNWRV4QVB5COFXHOSHOOMHIHHDYZFZPBDWN5PQNG7KA, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.341833061 (85h6m57.064396101s) [noise=true , address=104.36.82.4:4809 , piece_id=2SD7SIVSXJPQEPZXEK7354H3YFTRNA3T63NY7UMHTAZGPWMGXMJQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342059063 (85h6m57.064622113s) [noise=true , address=104.36.82.15:4260 , piece_id=MOUIJB5XJUIOWPHFS7OKBMT6W42BQEWMXANFUCKFMO4RBVPJLMJQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342084622 (85h6m57.064647662s) [noise=true , address=104.36.82.22:4509 , piece_id=6ADNMOVSC4NVUYYSNVR5XLRPV5POGVFFK7ZWKNWD2EGUVKZV3RMQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342403092 (85h6m57.064966132s) [noise=true , address=104.36.82.5:4842 , piece_id=JIRKR4WXYSVKRBADPJ7V6WLBNPTX4R55KQRN32MZUMV2T63WX5PA, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342409093 (85h6m57.064972133s) [noise=true , address=104.36.82.14:4240 , piece_id=C27DS6GK5LBLW6DNQ6OQMF4PVQZ753BFZQMK5K3UTDDZRNNTWR7Q, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342714468 (85h6m57.065277498s) [noise=true , address=104.36.82.46:4482 , piece_id=CI7ZROXKTZKIFFCGNMGSO35JKKXZYLPZKDGPWYDCDPHKLZTCVERQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342770655 (85h6m57.065333705s) [noise=true , address=79.127.201.215:4352 , piece_id=IKKYD4KSSEOCIEAXGMMQLUUG3CFH73E7E3SI5R7TI6NOBZXRCMKQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:47.342922767 (85h6m57.065485807s) [noise=true , address=104.36.82.54:4158 , piece_id=LD5WKHMVJVB7MPZ5LA64SNHJI32P3TZL4HK3NIUBPX3RLD3TLVDQ, seg_pos={0 0}, upload=81675]: Uploading piece... Jul 28 00:42:48.455780921 (85h6m58.178343961s) [upload=81675]: Next returned nil segment Jul 28 00:42:48.455802422 (85h6m58.178365472s) [upload=81675]: Waiting for error group managing segments... Jul 28 00:42:48.784431513 (85h6m58.506994564s) [noise=true , address=79.127.205.235:4688 , piece_id=TS2LM64VH7M7MRJJ2TND6WSBRD2UFIO5RQWHXL3BYPNTM4DQXIQQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: <nil> Jul 28 00:42:49.343244142 (85h6m59.065807213s) [noise=true , address=79.127.201.215:4352 , piece_id=IKKYD4KSSEOCIEAXGMMQLUUG3CFH73E7E3SI5R7TI6NOBZXRCMKQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: <nil> Jul 28 00:43:46.260330313 (85h7m55.982893353s) [noise=true , address=104.36.82.53:4148 , piece_id=VWEU2KT72MZRWNM2HHKUEX4ICI5OSZ4MHTDFHPDVTW3C4KWSX4KQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260423952 (85h7m55.982986992s) [noise=true , address=104.36.82.14:4240 , piece_id=C27DS6GK5LBLW6DNQ6OQMF4PVQZ753BFZQMK5K3UTDDZRNNTWR7Q, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:13SaC5StbguPT3abAT9oTX3mZAyGhJcpAA6b7ueyiDFHa8CCEo, address:104.36.82.14:4240): protocol: rpc: dial tcp 104.36.82.14:4240: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260482554 (85h7m55.983045594s) [noise=true , address=104.36.82.46:4482 , piece_id=CI7ZROXKTZKIFFCGNMGSO35JKKXZYLPZKDGPWYDCDPHKLZTCVERQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV, address:104.36.82.46:4482): protocol: rpc: dial tcp 104.36.82.46:4482: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260528673 (85h7m55.983091703s) [noise=true , address=104.36.82.3:4788 , piece_id=MZSH6QGBY67U6AX436UQQINE7654T2PALNLVENX63ISINDHH5EMQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:127EfVqSahKLLy8nF95kPd1a39xfThK4AKJ2X2XteKVCNntcMD4, address:104.36.82.3:4788): protocol: rpc: dial tcp 104.36.82.3:4788: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260550083 (85h7m55.983113123s) [noise=true , address=104.36.82.54:4158 , piece_id=LD5WKHMVJVB7MPZ5LA64SNHJI32P3TZL4HK3NIUBPX3RLD3TLVDQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: rpc: dial tcp 104.36.82.54:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260562578 (85h7m55.983125618s) [noise=true , address=104.36.82.15:4260 , piece_id=MOUIJB5XJUIOWPHFS7OKBMT6W42BQEWMXANFUCKFMO4RBVPJLMJQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1aLsgrV3k6vUSavgPkPGA2FF9HR3ryN9bnHutruFJHW1GYixeL, address:104.36.82.15:4260): protocol: rpc: dial tcp 104.36.82.15:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260594498 (85h7m55.983157538s) [noise=true , address=104.36.82.44:4891 , piece_id=7RKDP66EPDDTYVKLRDWVYM3JIUR3APDZGDHWXDDE2257UPBCNETA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1RHLM4MJmtKoHsUnsqccy1ehyD7zVSn6gnJNp7yJJmZhTJqPkU, address:104.36.82.44:4891): protocol: rpc: dial tcp 104.36.82.44:4891: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260599959 (85h7m55.983162989s) [noise=true , address=104.36.82.22:4509 , piece_id=6ADNMOVSC4NVUYYSNVR5XLRPV5POGVFFK7ZWKNWD2EGUVKZV3RMQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy, address:104.36.82.22:4509): protocol: rpc: dial tcp 104.36.82.22:4509: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260655365 (85h7m55.983218405s) [noise=true , address=104.36.82.13:4204 , piece_id=6JCSUI2O26UGR45QDIAQGKBW6W4I4GL5JSANEKA3WX5WBJ2PRVIA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: rpc: dial tcp 104.36.82.13:4204: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260672538 (85h7m55.983235578s) [noise=true , address=104.36.82.56:4253 , piece_id=V2U5R3XL4DVCM36KXZLIZ6AAWOVKHTOZQ36KLPSWYOHW7GF2PDGA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: rpc: dial tcp 104.36.82.56:4253: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260705721 (85h7m55.983268761s) [noise=true , address=104.36.82.55:4210 , piece_id=LXLEOZRR63IOBKCNUGKCL5HAQFMFP5WLYDY37JHTL5MZ5MEX3BTA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260710109 (85h7m55.983273149s) [noise=true , address=104.36.82.12:4158 , piece_id=XQZOWWS7LKNWRV4QVB5COFXHOSHOOMHIHHDYZFZPBDWN5PQNG7KA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:1SjM6Utm2YizeUbrBUD9Z7etE1nLadW5Fa1zYP7MioPeeqhdJ7, address:104.36.82.12:4158): protocol: rpc: dial tcp 104.36.82.12:4158: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260745348 (85h7m55.983308388s) [noise=true , address=104.36.82.10:4089 , piece_id=ED5J6MS3NPOT7LLN75PHJYT5NTWH43UUUFFALC3NDHSR4WTYD3CA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 188.166.225.170:47062->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260751860 (85h7m55.9833149s) [noise=true , address=104.36.82.5:4842 , piece_id=JIRKR4WXYSVKRBADPJ7V6WLBNPTX4R55KQRN32MZUMV2T63WX5PA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: write tcp 188.166.225.170:43612->104.36.82.5:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260760095 (85h7m55.983323135s) [noise=true , address=104.36.82.4:4809 , piece_id=2SD7SIVSXJPQEPZXEK7354H3YFTRNA3T63NY7UMHTAZGPWMGXMJQ, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:17LrYeCkrPWdiL9gGZ568PvfFct6cLVRQE4Qa92ykLgqXSfBL9, address:104.36.82.4:4809): protocol: rpc: dial tcp 104.36.82.4:4809: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260795423 (85h7m55.983358463s) [noise=true , address=104.36.82.4:4828 , piece_id=ISMIYHIB2GV5LUVOK5FV2NIQORBYIXOSEEPRP5EKG7W27LUFHPOA, seg_pos={0 0}, upload=81675]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:46250->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:43:46.260812075 (85h7m55.983375115s) [seg_pos={0 0}, upload=81675]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:43:46.490922522 (85h7m56.213485552s) [upload=81675]: Done uploading segments... Jul 28 00:43:46.490950155 (85h7m56.213513195s) [upload=81675]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.232693575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..X............X.......................X....................] duration: 1.868696121s FAIL (2025-07-28 00:43:43 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.868719375s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [....................X.......X.XX....X....X.......X.....X....] duration: 5.745423669s FAIL (2025-07-28 00:43:28 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.745453705s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X.X..........X.....X.X...........X.......] duration: 6.044680756s FAIL (2025-07-28 00:43:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.044710814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...X............X.......................X...................] duration: 1.897747454s FAIL (2025-07-28 00:43:18 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.897778634s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.....................X.......X.XX....X....X.......X.....X...] duration: 5.736130227s FAIL (2025-07-28 00:43:07 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.736156599s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [....X............X.......................X..................] duration: 1.893015371s FAIL (2025-07-28 00:43:02 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.893045677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X.X..........X.....X.X...........X......] duration: 5.767336717s FAIL (2025-07-28 00:42:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.76736408s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [.......................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.239594771s FAIL (2025-07-28 00:42:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 3) ======LOGS====== ce... Jul 28 00:41:39.003381710 (85h5m48.72594475s) [noise=true , address=79.127.201.215:4355 , piece_id=J4K4L3ZOPZR6YGS6E3YHPHP5QCFQU632SITGREMOGYWPSO3GD42Q, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003526477 (85h5m48.726089518s) [noise=true , address=104.36.82.14:4224 , piece_id=3CLJUK32K65AU5CTJBUORQKBDF7MJVPVF5S6CTVPWGASWZ7ZGWLA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003641417 (85h5m48.726204457s) [noise=true , address=104.36.82.11:4148 , piece_id=BOTXTYSNMNTURBEGQWMVIU4T7BZZVBSRL5FMUJIGBC6TJMCQUBBA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003743792 (85h5m48.726306832s) [noise=true , address=104.36.82.13:4191 , piece_id=V3EJ56GLXLTGUOJZH7GEPDH4HW4IPWTCWGUVYJXYZ2DXRUH6QKBA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.003912306 (85h5m48.726475356s) [noise=true , address=79.127.220.101:4045 , piece_id=XDCT3BR5WLV2ZY5PN74AOZQFXLK4ZJSGMIW6X37OBMECTTYKWIWQ, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.004316759 (85h5m48.726879799s) [noise=true , address=104.36.82.52:4094 , piece_id=P3K6JDUVTQTFRLGVETTRKXYF7DHWAGFGUBB7W7FY3IOKDWY5N62A, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.004597225 (85h5m48.727160265s) [noise=true , address=104.36.82.41:4788 , piece_id=CVBDBO3ATBQQNCZ2GXTTBETJTCJHYN5FY72VYHMD7CPSTV5I2X5A, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:39.005053248 (85h5m48.727616288s) [noise=true , address=104.36.82.42:4816 , piece_id=WQ3EASTEHG2BSNCWE62I4H3L5OZX35O37Q3IONF3CSYPAGQHN5QA, seg_pos={0 0}, upload=81659]: Uploading piece... Jul 28 00:41:40.406283592 (85h5m50.128846632s) [upload=81659]: Next returned nil segment Jul 28 00:41:40.406300505 (85h5m50.128863545s) [upload=81659]: Waiting for error group managing segments... Jul 28 00:41:40.828734633 (85h5m50.551297673s) [noise=true , address=79.127.220.101:4045 , piece_id=XDCT3BR5WLV2ZY5PN74AOZQFXLK4ZJSGMIW6X37OBMECTTYKWIWQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:41:42.498903904 (85h5m52.221466944s) [noise=true , address=79.127.201.215:4356 , piece_id=HPHM7PLAXQLE627NSVB54ZYJFS2D3AIHE56G4AOOQEISUGXND5YA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:41:51.457045956 (85h6m1.179609006s) [noise=true , address=79.127.201.215:4355 , piece_id=J4K4L3ZOPZR6YGS6E3YHPHP5QCFQU632SITGREMOGYWPSO3GD42Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: <nil> Jul 28 00:42:37.858869818 (85h6m47.581432858s) [noise=true , address=104.36.82.14:4224 , piece_id=3CLJUK32K65AU5CTJBUORQKBDF7MJVPVF5S6CTVPWGASWZ7ZGWLA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:128nLffK44ioiTpjWjAm2vHFTLouLZdw7BijqBAbsdjo99kPYNo, address:104.36.82.14:4224): protocol: rpc: dial tcp 104.36.82.14:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.858987814 (85h6m47.581550854s) [noise=true , address=104.36.82.54:4183 , piece_id=6KZME7RH5QJSS43BTZUHKSWT3GGS6IGCMTH46SJBDVEXC66WCAYA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1kovWN9SiUhcGQtZitjH9zgPj7iHaZwY5v37bjwPvArmgX44jA, address:104.36.82.54:4183): protocol: write tcp 188.166.225.170:34178->104.36.82.54:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859169261 (85h6m47.581732311s) [noise=true , address=104.36.82.41:4788 , piece_id=CVBDBO3ATBQQNCZ2GXTTBETJTCJHYN5FY72VYHMD7CPSTV5I2X5A, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1tJDCwfiqPgdc82nxvYrCiGB3KWijD4kSmYSAa8ZMBqRH2cA59, address:104.36.82.41:4788): protocol: write tcp 188.166.225.170:48552->104.36.82.41:4788: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859264493 (85h6m47.581827533s) [noise=true , address=104.36.82.50:4036 , piece_id=VBIQBIVTDWYTSTJK3LXO6357H56R3DGZLDU7TCBFBVXMOVZOLI5Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:17cebKoBWpDfVVnHqUDtFfJ71Lz8B7DqFonLumECeHxHf5o4sY, address:104.36.82.50:4036): protocol: rpc: dial tcp 104.36.82.50:4036: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859336050 (85h6m47.58189909s) [noise=true , address=104.36.82.8:4979 , piece_id=MYUANZUUFWMPM4GBPRG3723DTSLEX2UE7GSGU7FJ5PHT7YHK4D3Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12mjbYcXw9TCoRMkYNVzDYiT1VxMVnaQzdYxrRsBHD8ghy7GouC, address:104.36.82.8:4979): protocol: write tcp 188.166.225.170:45364->104.36.82.8:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859404742 (85h6m47.581967782s) [noise=true , address=104.36.82.13:4191 , piece_id=V3EJ56GLXLTGUOJZH7GEPDH4HW4IPWTCWGUVYJXYZ2DXRUH6QKBA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1sVWXyKPnLTobAhk6GXZTTP67FbyY3XJ1e2svTiUEa3rrghd2q, address:104.36.82.13:4191): protocol: write tcp 188.166.225.170:59394->104.36.82.13:4191: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859461381 (85h6m47.582024421s) [noise=true , address=104.36.82.52:4094 , piece_id=P3K6JDUVTQTFRLGVETTRKXYF7DHWAGFGUBB7W7FY3IOKDWY5N62A, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 188.166.225.170:46340->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859867066 (85h6m47.582430116s) [noise=true , address=104.36.82.42:4816 , piece_id=WQ3EASTEHG2BSNCWE62I4H3L5OZX35O37Q3IONF3CSYPAGQHN5QA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1nq6pTeLCMEmtagczJkTMAdpADG5JrbSKhUjXAHy9oeS1pTpSx, address:104.36.82.42:4816): protocol: write tcp 188.166.225.170:53552->104.36.82.42:4816: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.859955466 (85h6m47.582518516s) [noise=true , address=104.36.82.53:4133 , piece_id=QGV4FYPXFALP4AUPUEMJIATIZ6Q2G7NWHEQU24WQYOCXFEFHHS4Q, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: write tcp 188.166.225.170:32996->104.36.82.53:4133: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860042943 (85h6m47.582605993s) [noise=true , address=104.36.82.55:4221 , piece_id=FLJIATJ6PKQYCLWCMQPUVX2NIN6UFOWLUYEO4MDE3D47FFIYPVWA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: write tcp 188.166.225.170:33910->104.36.82.55:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860129338 (85h6m47.582692378s) [noise=true , address=104.36.82.21:4497 , piece_id=LM5DLP2VY2GJMIBWIPPPRLMHLCAA3IB7Q4ZF273NRIA4TGTMSJAA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: write tcp 188.166.225.170:35874->104.36.82.21:4497: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860152032 (85h6m47.582715082s) [noise=true , address=104.36.82.56:4249 , piece_id=ENQ5BFW7XGHUBVNQT7XFKG2QW2YBGP2CQLEEMPUGLJ4CXVQKK3YQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1LuzmiCefHA9kX3HHWHqviHfG4Ts2t5DYu5nfJRWquaQ8nQrzx, address:104.36.82.56:4249): protocol: write tcp 188.166.225.170:60768->104.36.82.56:4249: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860210534 (85h6m47.582773574s) [noise=true , address=104.36.82.9:4023 , piece_id=6EXTQEATJJPFCFK35VUT25UHRYLGKYKSRMVOAXJEPIEMW3LSH3DQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:1j2S9tgWpLwmyvXZSq2qKKcbaC8r7LkzfDCaMxCejAni4CST1g, address:104.36.82.9:4023): protocol: write tcp 188.166.225.170:57586->104.36.82.9:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860250580 (85h6m47.58281363s) [noise=true , address=104.36.82.11:4148 , piece_id=BOTXTYSNMNTURBEGQWMVIU4T7BZZVBSRL5FMUJIGBC6TJMCQUBBA, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12Rkh9uZUA5jXBVZvvt1jE7GHiEszKvR9PweKEry2AkdwoCiANs, address:104.36.82.11:4148): protocol: write tcp 188.166.225.170:43986->104.36.82.11:4148: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860256863 (85h6m47.582819903s) [noise=true , address=104.36.82.12:4183 , piece_id=7UPVLWN5KN6VMYKRFF3S3AD6A67TGVR4BA454FY6RJQ4BBS7OWCQ, seg_pos={0 0}, upload=81659]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: write tcp 188.166.225.170:56608->104.36.82.12:4183: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:42:37.860274676 (85h6m47.582837716s) [seg_pos={0 0}, upload=81659]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil> Jul 28 00:42:38.096205640 (85h6m47.81876868s) [upload=81659]: Done uploading segments... Jul 28 00:42:38.096230938 (85h6m47.818793978s) [upload=81659]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3) Duration: 1m0.239607084s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X.......X.XX....X....X.......X.....X..] duration: 5.779685161s FAIL (2025-07-28 00:42:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.779713937s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....X............X.......................X.................] duration: 1.896177581s FAIL (2025-07-28 00:42:12 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.896203682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [......X............X.......................X................] duration: 1.902463939s FAIL (2025-07-28 00:41:39 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.902488435s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.230841955s FAIL (2025-07-28 00:41:37 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== 5VBSJQIAD6KT7TA, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.483904554 (85h4m48.206467584s) [noise=true , address=104.36.82.47:4532 , piece_id=ZUXRGJEKLS5BN6N2F2VJACTGSDCKE5AGDH6M4HAMIZEFYT4A3Z5A, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484095570 (85h4m48.20665861s) [noise=true , address=104.36.82.35:4611 , piece_id=4BOT7PNPJMBFDZXOZVMEML723KJ7RZ5EYANGGD3N4L2KMRYICLLA, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484291685 (85h4m48.206854715s) [noise=true , address=104.36.82.41:4765 , piece_id=NJNR7ZP4AHPM7QHLFHV3ZMVKH5C7EH7PZ5B3D2DKSQLNSZXYUOUQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484528588 (85h4m48.207091628s) [noise=true , address=104.36.82.52:4094 , piece_id=MBEKPZYUK625WWXP2KIP3BFV6POQJDFRG4MQZWZNHYUGF4YXB5LQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484733320 (85h4m48.20729636s) [noise=true , address=104.36.82.12:4159 , piece_id=2MHN6XSDHHCCUUQQQJQ2MFO5GGWZAOVTUF3STXGG7ZAXTQSJEZWQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.484932951 (85h4m48.207495991s) [noise=true , address=104.36.82.14:4244 , piece_id=BHAGCPT6S6GL6FTE2QYHFEKBC6HR5O7A42OQNPVBXYHLWJGYJABQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.485117154 (85h4m48.207680194s) [noise=true , address=104.36.82.15:4266 , piece_id=ZPNNDC26LLPZZLOKNHDNSSDX3BUUQTQOH2YBPM5LD5T5VG3CFAUQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:40:38.485299373 (85h4m48.207862413s) [noise=true , address=104.36.82.53:4154 , piece_id=NYQSCKB5E326IZKFU6ARQM4E4AJWOC34T3J3CNN2DY4QLULN74FQ, seg_pos={0 0}, upload=81644]: Uploading piece... Jul 28 00:41:37.397645517 (85h5m47.120208557s) [upload=81644]: Next segment err: context deadline exceeded Jul 28 00:41:37.398329837 (85h5m47.120892867s) [noise=true , address=104.36.82.35:4611 , piece_id=4BOT7PNPJMBFDZXOZVMEML723KJ7RZ5EYANGGD3N4L2KMRYICLLA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1zkqEMSKx7zSHPFNZ51b5TLzL2cFQL8VyxAG14Lg1pT8zAu3tc, address:104.36.82.35:4611): protocol: rpc: dial tcp 104.36.82.35:4611: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398370724 (85h5m47.120933764s) [noise=true , address=104.36.82.41:4765 , piece_id=NJNR7ZP4AHPM7QHLFHV3ZMVKH5C7EH7PZ5B3D2DKSQLNSZXYUOUQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12SRRQBu9efFwqpKMvLBQhaZ9s3XQXVekdmefCBXuQPgRdik2QT, address:104.36.82.41:4765): protocol: rpc: dial tcp 104.36.82.41:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398400933 (85h5m47.120963973s) [noise=true , address=104.36.82.47:4536 , piece_id=HP4UF6UP6FURKBUEDANDS7JTFU4XIRITAEUKQ6UUZ4XICOC2GEVQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1WoVNAp5RfVviaxN4RWCUDEhxLKsdGB8bgE6gcUcgZKcM4Cz1s, address:104.36.82.47:4536): protocol: write tcp 188.166.225.170:43872->104.36.82.47:4536: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398433756 (85h5m47.120996786s) [noise=true , address=104.36.82.9:4026 , piece_id=XS3MZD67O24ELSDA7KLSZEA7ELBPU5ZU3CZV5WRURGZZ2WJMJVGQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1M7HJBLoxHkRHXbPuDiqwxdb4EAq5wktfJY5FygdoDdgy59rRq, address:104.36.82.9:4026): protocol: rpc: dial tcp 104.36.82.9:4026: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398461869 (85h5m47.121024909s) [noise=true , address=104.36.82.52:4094 , piece_id=MBEKPZYUK625WWXP2KIP3BFV6POQJDFRG4MQZWZNHYUGF4YXB5LQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398495073 (85h5m47.121058113s) [noise=true , address=104.36.82.12:4159 , piece_id=2MHN6XSDHHCCUUQQQJQ2MFO5GGWZAOVTUF3STXGG7ZAXTQSJEZWQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1taQkf86SDCwKJaXVcfu1YcSNz8SfpAtChruvcZVFfy55qZdtX, address:104.36.82.12:4159): protocol: rpc: dial tcp 104.36.82.12:4159: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398526703 (85h5m47.121089743s) [noise=true , address=104.36.82.14:4244 , piece_id=BHAGCPT6S6GL6FTE2QYHFEKBC6HR5O7A42OQNPVBXYHLWJGYJABQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:129iVGvtfYzqdPJk52ef8uGNHKp1jJtr2MGE4uXq8oKQmmXX6C, address:104.36.82.14:4244): protocol: write tcp 188.166.225.170:60544->104.36.82.14:4244: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398565909 (85h5m47.121128949s) [noise=true , address=104.36.82.15:4266 , piece_id=ZPNNDC26LLPZZLOKNHDNSSDX3BUUQTQOH2YBPM5LD5T5VG3CFAUQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j, address:104.36.82.15:4266): protocol: rpc: dial tcp 104.36.82.15:4266: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398584574 (85h5m47.121147614s) [noise=true , address=104.36.82.42:4821 , piece_id=NZA2C62FVEJ7JM74GIKAQU77XIVW63NUJ26DCDCWIRLD2GSTOSVQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398630552 (85h5m47.121193592s) [noise=true , address=104.36.82.47:4532 , piece_id=ZUXRGJEKLS5BN6N2F2VJACTGSDCKE5AGDH6M4HAMIZEFYT4A3Z5A, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398659948 (85h5m47.121222988s) [noise=true , address=104.36.82.56:4259 , piece_id=PYVSMYNYJU5RTL4X3NBYRF27NRWQ3XPJL6ICU2IXCM6RO35WPDBQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12BCJsPfe1NNUxVUfQWKbvkzviQWYgMxkRZjDhKJ7KFZc1GmnjV, address:104.36.82.56:4259): protocol: rpc: dial tcp 104.36.82.56:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398683944 (85h5m47.121246974s) [noise=true , address=104.36.82.13:4197 , piece_id=UAYGPQPDFZIYM6M4VZR24HF42CUHXNAKCHTSU6FNKCSBHEGGPTMA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1cB89C5xnya9WGD9dGBsSouqkZH3nLqytrg9Qo4UH2kdFonTjM, address:104.36.82.13:4197): protocol: rpc: dial tcp 104.36.82.13:4197: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398688813 (85h5m47.121251843s) [noise=true , address=104.36.82.54:4185 , piece_id=QEGMEMSFPEGRDOREOT6ZUW4LDZGT7EIUEO6YO5VBSJQIAD6KT7TA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1cLdyyVLrp1yBmgsfBrfYc6N2RFbBichYoBbo2F7EZfaBXtkfk, address:104.36.82.54:4185): protocol: rpc: dial tcp 104.36.82.54:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398720634 (85h5m47.121283664s) [noise=true , address=104.36.82.10:4098 , piece_id=IDQFW3G3EAQZ52ZB77LX44TEWXJGCOHURAD3D2ESUJYRA4QMENRQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:122fmWY53WxM6KHuXzdKsCbWWCtCFDUGowvSqALWUdFnbYVJoSY, address:104.36.82.10:4098): protocol: rpc: dial tcp 104.36.82.10:4098: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398732848 (85h5m47.121295878s) [noise=true , address=104.36.82.4:4830 , piece_id=P56LXY4ZIMBUGXUEGJ6DLFRS3QUTABZ4PP6W77FHPXDN24E36OXQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1p91PtS64Ue8JkPmbh2f3gig8cTQ3AMdDJPaP1249h4XmXUJrp, address:104.36.82.4:4830): protocol: rpc: dial tcp 104.36.82.4:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398739220 (85h5m47.12130226s) [noise=true , address=104.36.82.44:4870 , piece_id=2X6TTJTEUIRPUGE75YEIO66LJFV5JQ35WPLTKN6KA3NFTLQSAPRA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt, address:104.36.82.44:4870): protocol: rpc: dial tcp 104.36.82.44:4870: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398744019 (85h5m47.121307059s) [noise=true , address=104.36.82.53:4154 , piece_id=NYQSCKB5E326IZKFU6ARQM4E4AJWOC34T3J3CNN2DY4QLULN74FQ, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: rpc: dial tcp 104.36.82.53:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398751233 (85h5m47.121314273s) [noise=true , address=104.36.82.5:4864 , piece_id=HLG6U6REL3VKPA5FQIPUI3LI5WWVXXRG6RYL5LT6XQBJMIZL36QA, seg_pos={0 0}, upload=81644]: Done uploading piece. err: ecclient: upload failed (node:12XXH8TYK6Qfma867943QYYXycyL6osQhcQYEjdMdJVTUDfda2u, address:104.36.82.5:4864): protocol: write tcp 188.166.225.170:51450->104.36.82.5:4864: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:41:37.398768416 (85h5m47.121331446s) [seg_pos={0 0}, upload=81644]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.230855491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X.X..........X.....X.X...........X.....] duration: 5.819633605s FAIL (2025-07-28 00:41:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.8196572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X.......X.XX....X....X.......X.....X.] duration: 5.885188643s FAIL (2025-07-28 00:41:21 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.885217196s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......X............X.......................X...............] duration: 1.850004538s FAIL (2025-07-28 00:41:12 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.850039105s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X.......X.XX....X....X.......X.....X] duration: 30.238408019s FAIL (2025-07-28 00:40:50 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.238430632s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [.........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.232923955s FAIL (2025-07-28 00:40:37 UTC) orbiter warning: ======ERROR====== uplink: encryption: failed to upload enough pieces (needed at least 14 but got 0) ======LOGS====== ess=104.36.82.16:4313 , piece_id=EZC2I757SSU77IGFOJ62TI5OA4CPD3LZYSU5GAPQZ3KQCE6C7NOA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031296221 (85h3m47.753859261s) [noise=true , address=104.36.82.54:4169 , piece_id=MTILMOCHCF47BKDWJ3BEJSGKT3FM5AETFKO3MM5726DUI6SQANQA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031408105 (85h3m47.753971135s) [noise=true , address=104.36.82.11:4120 , piece_id=UUEIVUSY7OSDGL6V6MYA7BMMJQ6Z6DTWZNHC72M57JZLS3TDSPEA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031682440 (85h3m47.75424564s) [noise=true , address=104.36.82.46:4509 , piece_id=DD3H22UHCU2NUQB3I7GNUKSNTYK2SYVPBI7HHOHWPQVXJHVI3SGA, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.031692389 (85h3m47.754255429s) [noise=true , address=104.36.82.44:4886 , piece_id=UVFREUL5IVXMTEUALTEP4Z46RY6XOGGZSYX3CZ73DKRRVNP272DQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.032495757 (85h3m47.755058797s) [noise=true , address=104.36.82.4:4828 , piece_id=OB2O5EOVBUDNAS67MZS5SJHMWZRRYGJE6GMJ3BMLRKGM3MISX2LQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:39:38.032924907 (85h3m47.755487947s) [noise=true , address=104.36.82.10:4101 , piece_id=2GNAF6R4VL7IP4QZHH6QMKBLRVZ56XSZYX5ZMPYG4XH7XJK4D2AQ, seg_pos={0 0}, upload=81625]: Uploading piece... Jul 28 00:40:36.944988797 (85h4m46.667551847s) [upload=81625]: Next segment err: context deadline exceeded Jul 28 00:40:36.945542306 (85h4m46.668105346s) [noise=true , address=104.36.82.56:4236 , piece_id=OHP5ABJYYV4BVTGCZVIJLYRCEZSH2ZKSFAN6IQVUJCNQRVUWRBMA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1pRLRqP5JEepqEKV9R3BUUhztQXnakq8QKMNjBZ1KR5TMMAXAb, address:104.36.82.56:4236): protocol: write tcp 188.166.225.170:36016->104.36.82.56:4236: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945582442 (85h4m46.668145482s) [noise=true , address=104.36.82.55:4200 , piece_id=2J4PE7XQTCE6QPB3AC4RDBIJ2VAS6T4FOILNQVUH7D3YZHD73FWA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1Y936LLJepKpXAF6huSDRd61pWHNmdntRVNZutbadfeaZerMyj, address:104.36.82.55:4200): protocol: rpc: dial tcp 104.36.82.55:4200: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945610476 (85h4m46.668173516s) [noise=true , address=104.36.82.47:4535 , piece_id=HYAE6ZMAGOJTI4Q563X6HB6DIAEIRJYDQ4AKDNLWD232RSP45THQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945652727 (85h4m46.668215767s) [noise=true , address=104.36.82.14:4246 , piece_id=VNGLR3VR4LZB2V5AR2JH5BFSIN4FCXY2Y2Y5UFNL4SE5ENFHWNAA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12dqLB6LVQJaSdEfjKmu5nXqRST733hn5rTxCcMQS8YJVv8Hwdc, address:104.36.82.14:4246): protocol: write tcp 188.166.225.170:58426->104.36.82.14:4246: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945685009 (85h4m46.668248049s) [noise=true , address=104.36.82.54:4169 , piece_id=MTILMOCHCF47BKDWJ3BEJSGKT3FM5AETFKO3MM5726DUI6SQANQA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: write tcp 188.166.225.170:52136->104.36.82.54:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945720085 (85h4m46.668283125s) [noise=true , address=104.36.82.11:4120 , piece_id=UUEIVUSY7OSDGL6V6MYA7BMMJQ6Z6DTWZNHC72M57JZLS3TDSPEA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:123jcA328VCSUaWMKvSaeXip1U6Sc4SSvuRaoHF8fvsF6Tt6DHW, address:104.36.82.11:4120): protocol: rpc: dial tcp 104.36.82.11:4120: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945769600 (85h4m46.66833265s) [noise=true , address=104.36.82.10:4101 , piece_id=2GNAF6R4VL7IP4QZHH6QMKBLRVZ56XSZYX5ZMPYG4XH7XJK4D2AQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1RmTDQrWoQBSpHrcUsb2PYijpcFdCyRya7VrMkxhWeQXdGeQpp, address:104.36.82.10:4101): protocol: write tcp 188.166.225.170:45392->104.36.82.10:4101: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.945824155 (85h4m46.668387195s) [noise=true , address=104.36.82.43:4857 , piece_id=GPSOWXYUBBNEIBFDNCNC22P36RFLTNAAQG4PUJUKSEJB54XLWUXA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1DK5cJhprAFbNCSceZ527v5rMFw7e3teUTsxBXVPcjXUDiVv4P, address:104.36.82.43:4857): protocol: write tcp 188.166.225.170:38522->104.36.82.43:4857: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946087538 (85h4m46.668650578s) [noise=true , address=104.36.82.12:4167 , piece_id=LIZXSRCIVBZWXPAKNL2LZYQUK26UTEGIQOWW3BHRRSQFE6VJ25ZQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih, address:104.36.82.12:4167): protocol: rpc: dial tcp 104.36.82.12:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946134198 (85h4m46.668697238s) [noise=true , address=104.36.82.47:4532 , piece_id=4N5ZM6ADCNWF3TMZ7A555WYURY4RGAKQ6OTOJTALZ2T5RQU3XG2Q, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946178353 (85h4m46.668741393s) [noise=true , address=104.36.82.53:4121 , piece_id=KA2RIAAOMBPF5N47RSL44R2L6L2PXISD7IUQUXRHLUAHHCRBEF3A, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:104.36.82.53:4121): protocol: write tcp 188.166.225.170:57456->104.36.82.53:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946216084 (85h4m46.668779124s) [noise=true , address=104.36.82.44:4886 , piece_id=UVFREUL5IVXMTEUALTEP4Z46RY6XOGGZSYX3CZ73DKRRVNP272DQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 188.166.225.170:49626->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946248606 (85h4m46.668811646s) [noise=true , address=104.36.82.47:4545 , piece_id=N767D2NVW22SIHYCCMPBIS3RG75CSUBR4ABESUXGWJPTU7PN5X2A, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: write tcp 188.166.225.170:37190->104.36.82.47:4545: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946409735 (85h4m46.668972775s) [noise=true , address=104.36.82.4:4828 , piece_id=OB2O5EOVBUDNAS67MZS5SJHMWZRRYGJE6GMJ3BMLRKGM3MISX2LQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12MV49LDqJJn8S7nLwgVSpaYiGKa3BZJYbbge8oRuiXQfbNecaJ, address:104.36.82.4:4828): protocol: write tcp 188.166.225.170:36886->104.36.82.4:4828: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946456244 (85h4m46.669019284s) [noise=true , address=104.36.82.13:4218 , piece_id=MVZWT5N3CS4XCLCJCAM7BMZYFZHPQYGVNHUQU73GB6YNITTCM74Q, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12mbzxg1BjmbXeHxGgheZmQoN91k8NgaAUUASPkJe3fEmS4eEkc, address:104.36.82.13:4218): protocol: write tcp 188.166.225.170:39014->104.36.82.13:4218: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946504437 (85h4m46.669067477s) [noise=true , address=104.36.82.5:4842 , piece_id=3IHSL6RN7IV2K4XGODMY47543RKRHNT6OC7KZIUI3GMGYU3EWKNQ, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: write tcp 188.166.225.170:52642->104.36.82.5:4842: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.946848965 (85h4m46.669412005s) [noise=true , address=104.36.82.16:4313 , piece_id=EZC2I757SSU77IGFOJ62TI5OA4CPD3LZYSU5GAPQZ3KQCE6C7NOA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1nJzsvcbhmuGGQyUHPVALuTiwt2rmrtRkEY8R1P1AfsUr33kDA, address:104.36.82.16:4313): protocol: rpc: dial tcp 104.36.82.16:4313: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:40:36.947176843 (85h4m46.669739883s) [noise=true , address=104.36.82.46:4509 , piece_id=DD3H22UHCU2NUQB3I7GNUKSNTYK2SYVPBI7HHOHWPQVXJHVI3SGA, seg_pos={0 0}, upload=81625]: Done uploading piece. err: ecclient: upload failed (node:1y2NmauNvCyrMJ5z66Lihb3h3C9nwdJYqfSfNmC5dwDGYQqUKh, address:104.36.82.46:4509): protocol: rpc: dial tcp 104.36.82.46:4509: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 28 00:40:36.947196170 (85h4m46.6697592s) [seg_pos={0 0}, upload=81625]: Done waiting for segment. successful: 0 optimal: 14 errs: <nil> Duration: 1m0.232937061s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X.X..........X.....X.X...........X....] duration: 5.910255022s FAIL (2025-07-28 00:40:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.910286762s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [........X............X.......................X..............] duration: 1.810646015s FAIL (2025-07-28 00:40:32 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.810681062s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.........................X.......X.XX....X....X.......X.....] duration: 5.885959664s FAIL (2025-07-28 00:39:57 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.885982738s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X.X..........X.....X.X...........X...] duration: 5.777641359s FAIL (2025-07-28 00:39:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.77766315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.........X............X.......................X.............] duration: 1.836720474s FAIL (2025-07-28 00:39:40 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.836744711s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [..........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231953221s FAIL (2025-07-28 00:39:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== , piece_id=6YJ3ALZ7BUNNKTK5KNDXEWUVXVLTDXTPBFO7QROA5KA7UFEZVH6Q, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.591873129 (85h2m47.314436169s) [noise=true , address=77.90.31.16:4744 , piece_id=4ZQMZ27OA6POWWEI2QKO67YOFXXYJKSNTHAQZGZY2H4725XZBH5A, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.592119180 (85h2m47.31468222s) [noise=true , address=104.36.82.42:4797 , piece_id=5KVVRBI25LHWAV3YBSQJYQ4E5EQTHSK5B4AAU2IFGZQMCNMIPPDA, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.592309414 (85h2m47.314872454s) [noise=true , address=104.36.82.3:4800 , piece_id=UW7XJFWDEZUPOOOXJIJBKPXTZXGOQZWDH3V5B36TCW3C4NMF4RRA, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.592584841 (85h2m47.315147881s) [noise=true , address=104.36.82.56:4261 , piece_id=HVPSSS2B6XADE2CTMBJZLCBCSTESWU7HMSZKBZWKUESIYTR7TU5A, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.592780274 (85h2m47.315343314s) [noise=true , address=104.36.82.46:4481 , piece_id=W735WRYS4FVGHUFQR4K263PUVQVMSXZEVL5BDV5ZJMVO2FKTHNRQ, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.593017128 (85h2m47.315580158s) [noise=true , address=104.36.82.53:4127 , piece_id=EKZKVF6TXHW2HA67EJI637LRLU3GJY5JI3JE32WBUQXKXMB4N3BA, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:37.593270192 (85h2m47.315833232s) [noise=true , address=104.36.82.21:4500 , piece_id=FVVX5PMXOPSS2LZGE3TMDUZU2UB4OF3T5Y3YT2QJ3PFFVLBTOQ2Q, seg_pos={0 0}, upload=81611]: Uploading piece... Jul 28 00:38:41.453639288 (85h2m51.176202338s) [upload=81611]: Next returned nil segment Jul 28 00:38:41.453658475 (85h2m51.176221515s) [upload=81611]: Waiting for error group managing segments... Jul 28 00:38:42.669407204 (85h2m52.391970244s) [noise=true , address=77.90.31.16:4744 , piece_id=4ZQMZ27OA6POWWEI2QKO67YOFXXYJKSNTHAQZGZY2H4725XZBH5A, seg_pos={0 0}, upload=81611]: Done uploading piece. err: <nil> Jul 28 00:39:36.488877495 (85h3m46.211440535s) [noise=true , address=104.36.82.47:4546 , piece_id=ERLYKAMORADOAO3RI7VCXIT4J7LOYYOPPG7YBTKPWOIGUFJ4GSCQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1FJGmqUskGmxYctVCEaYhRHq98m1VCVZiWL1ShEMJzmieaYo1u, address:104.36.82.47:4546): protocol: write tcp 188.166.225.170:38196->104.36.82.47:4546: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.488924656 (85h3m46.211487696s) [noise=true , address=104.36.82.40:4947 , piece_id=HRJJBENZQIZXZJMV5L67IA6SK2UA7XFX4TOIYI6TDPOB2N6V4JNA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1vVHmSRAfugvyvTkEZHitE4uFMGXxgrbnWfJuGD9MzS9avAYsU, address:104.36.82.40:4947): protocol: rpc: dial tcp 104.36.82.40:4947: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.488951156 (85h3m46.211514196s) [noise=true , address=104.36.82.52:4116 , piece_id=U63BKDMAQBB43R63H2HTHAFSZG6IYMSVKAX5KPNDS6XHJUK4TYJQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12vKCXgWTAsipTvUK9XA3sL6DznFA8BqZc6WCN84PBuLhKCTT9m, address:104.36.82.52:4116): protocol: rpc: dial tcp 104.36.82.52:4116: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.488964842 (85h3m46.211527882s) [noise=true , address=104.36.82.55:4207 , piece_id=BG6USRTMG6YKGAZBKSDSOZWX2TRTVG7HTR6FTCIAZNJ2ZOM7EAZQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.488985091 (85h3m46.211548131s) [noise=true , address=104.36.82.54:4176 , piece_id=NMYZEHLODKQEORJOKMC7IVCFMFZWS63TCCPKOLHIYNDK5ULTBUGQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1Yi5AibNiCNV8vUb9MKgooHCw8Nr79m9HqTrkrYybNZmatg9PD, address:104.36.82.54:4176): protocol: rpc: dial tcp 104.36.82.54:4176: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.488996432 (85h3m46.211559472s) [noise=true , address=104.36.82.11:4139 , piece_id=D67GJVUWC5TLK5AYHRGA3ZS2WF5EMHCP7UWK4MSYVUCUJOIIDILA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1EiZjV2QMYue3KQDsNRhhdGpCQjybDv54wTuXsJTNtyeV8hHeh, address:104.36.82.11:4139): protocol: rpc: dial tcp 104.36.82.11:4139: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489027101 (85h3m46.211590141s) [noise=true , address=104.36.82.16:4310 , piece_id=6YJ3ALZ7BUNNKTK5KNDXEWUVXVLTDXTPBFO7QROA5KA7UFEZVH6Q, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1g14evoaeUnhJThf5QKbQT1Gd8WW9MHvaYQhQ1GRgwPfDLoC5R, address:104.36.82.16:4310): protocol: rpc: dial tcp 104.36.82.16:4310: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489055947 (85h3m46.211618987s) [noise=true , address=104.36.82.3:4800 , piece_id=UW7XJFWDEZUPOOOXJIJBKPXTZXGOQZWDH3V5B36TCW3C4NMF4RRA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12CWPcNRnN5pX4mAU6Zkbwgy2teaAfusbmeH7niPaY9r2yiDLtT, address:104.36.82.3:4800): protocol: rpc: dial tcp 104.36.82.3:4800: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489082216 (85h3m46.211645256s) [noise=true , address=104.36.82.42:4797 , piece_id=5KVVRBI25LHWAV3YBSQJYQ4E5EQTHSK5B4AAU2IFGZQMCNMIPPDA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1VwwhXQqXCHcgP37bcQAUA6urv5YGVt7NCpRNrqMKSSE5AScK9, address:104.36.82.42:4797): protocol: rpc: dial tcp 104.36.82.42:4797: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489094130 (85h3m46.21165717s) [noise=true , address=104.36.82.53:4127 , piece_id=EKZKVF6TXHW2HA67EJI637LRLU3GJY5JI3JE32WBUQXKXMB4N3BA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489151900 (85h3m46.21171494s) [noise=true , address=104.36.82.12:4166 , piece_id=PX7XE53BEHI23ECCD22EVQX6C6J3CWJCDSJXGF5VZY7DRWSW4H4Q, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12iCB7c4ZfpqafpD1PtpbE6BfU81wJW65LmrPVJXtD58yeuoR9t, address:104.36.82.12:4166): protocol: rpc: dial tcp 104.36.82.12:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489216113 (85h3m46.211779153s) [noise=true , address=104.36.82.41:4795 , piece_id=BFDCFDLYROGE7WTWX6GKPX44ZJ5YVH3K47SNZSWQ57JKXSYLRVMA, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12ovwD5mScxxBf7RtRjAbBqRiqgAmTC8ajN3oBAXSaWrDQ1t9xk, address:104.36.82.41:4795): protocol: write tcp 188.166.225.170:52092->104.36.82.41:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489229258 (85h3m46.211792288s) [noise=true , address=104.36.82.56:4261 , piece_id=HVPSSS2B6XADE2CTMBJZLCBCSTESWU7HMSZKBZWKUESIYTR7TU5A, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489251922 (85h3m46.211814962s) [noise=true , address=104.36.82.44:4902 , piece_id=G437LTL3BOKTYDUSUMX5GSD2DWWNBLE6H6JOGFYI44YUM2O4DR4A, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1Be9T5kXUwLU3C6wBeKFV3Q7aZFbZDTNG1oaEft8V6rmySUcEE, address:104.36.82.44:4902): protocol: write tcp 188.166.225.170:32770->104.36.82.44:4902: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489271719 (85h3m46.211834759s) [noise=true , address=104.36.82.13:4210 , piece_id=GFGPOU2V4NLO3AGCH6IPO3IEYPZBIE2VS3XSAZ74XFZLRVWMHMBQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1ybrcx8SPgyEsJWq2P1KYaTwTj5v5BfENKNEZZxaKRvKvuqTiL, address:104.36.82.13:4210): protocol: rpc: dial tcp 104.36.82.13:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489280256 (85h3m46.211843296s) [noise=true , address=104.36.82.21:4500 , piece_id=FVVX5PMXOPSS2LZGE3TMDUZU2UB4OF3T5Y3YT2QJ3PFFVLBTOQ2Q, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:1FPAiRym6MSHqcrk5PMK8sCafNt3sv55TrS848QJmBaUpzDcsx, address:104.36.82.21:4500): protocol: rpc: dial tcp 104.36.82.21:4500: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489296657 (85h3m46.211859687s) [noise=true , address=104.36.82.46:4481 , piece_id=W735WRYS4FVGHUFQR4K263PUVQVMSXZEVL5BDV5ZJMVO2FKTHNRQ, seg_pos={0 0}, upload=81611]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 188.166.225.170:60940->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:39:36.489316445 (85h3m46.211879485s) [seg_pos={0 0}, upload=81611]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:39:36.718266605 (85h3m46.440829635s) [upload=81611]: Done uploading segments... Jul 28 00:39:36.718292735 (85h3m46.440855775s) [upload=81611]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.231970124s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X.X..........X.....X.X...........X..] duration: 6.25409944s FAIL (2025-07-28 00:39:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.25413025s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..........................X.......X.XX....X....X.......X....] duration: 5.726290769s FAIL (2025-07-28 00:39:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.726322881s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X.X..........X.....X.X...........X.] duration: 6.012950695s FAIL (2025-07-28 00:38:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.012988436s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..........X............X.......................X............] duration: 1.857621803s FAIL (2025-07-28 00:38:38 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.857657543s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X.X..........X.....X.X...........X] duration: 30.239590852s FAIL (2025-07-28 00:38:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ess=45.91.116.13:4155 , piece_id=N2IRR7LH27DX75AZIGOQGPW4BDGFI7CMRKA24M2UYKNB2TORR76Q, seg_pos={0 0}, upload=81590]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Jul 28 00:37:09.814987852 (85h1m19.537550892s) [seg_pos={0 0}, upload=81590]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:37:09.815062484 (85h1m19.537625524s) [upload=81590]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:37:09.815066463 (85h1m19.537629503s) [upload=81590]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:37:10.095114538 (85h1m19.817677578s) [upload=81590]: Done uploading segments... Jul 28 00:37:10.095134416 (85h1m19.817697446s) [upload=81590]: Upload finished. err: <nil> Jul 28 00:38:02.738662065 (85h2m12.461225135s) [upload=81599]: Starting upload Jul 28 00:38:02.738787566 (85h2m12.461350636s) [upload=81599]: Done starting upload Jul 28 00:38:02.738929106 (85h2m12.461492177s) [upload=81599]: Uploading segments... Jul 28 00:38:02.739135141 (85h2m12.461698201s) [upload=81599]: Got next segment. Inline: false Jul 28 00:38:02.739200586 (85h2m12.461763656s) [upload=81599]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:38:02.739208401 (85h2m12.461771491s) [upload=81599]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:38:03.190604705 (85h2m12.913167776s) [seg_pos={0 0}, upload=81599]: Begin upload segment... Jul 28 00:38:03.190668337 (85h2m12.913231397s) [seg_pos={0 0}, upload=81599]: Done begin upload segment. Jul 28 00:38:03.190709426 (85h2m12.913272516s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190720948 (85h2m12.913283988s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.190782266 (85h2m12.913345306s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191494367 (85h2m12.914057407s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.191514155 (85h2m12.914077216s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192155223 (85h2m12.914718303s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192296152 (85h2m12.914859212s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192672862 (85h2m12.915235912s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.192793383 (85h2m12.915356434s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193135357 (85h2m12.915698427s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193501498 (85h2m12.916064548s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.193897396 (85h2m12.916460436s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.194266802 (85h2m12.916829852s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.195223863 (85h2m12.917786913s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196233425 (85h2m12.918796465s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:03.196874782 (85h2m12.919437822s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Uploading piece... Jul 28 00:38:04.827971375 (85h2m14.550534415s) [upload=81599]: Next returned nil segment Jul 28 00:38:04.827982988 (85h2m14.550546018s) [upload=81599]: Waiting for error group managing segments... Jul 28 00:38:05.207136591 (85h2m14.929699631s) [noise=true , address=77.90.31.12:4590 , piece_id=TWTFXYQCMKOBTQHDSSH77XU4NIRVGECUOT7EIZPB5BTAR62PMTTQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.235830782 (85h2m14.958393822s) [noise=true , address=45.91.116.4:4939 , piece_id=Z6EWOXDGN2RS3PIRSRS63PER3QHSKBGO25R6UAMKO44E7LUK2C4A, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.281751633 (85h2m15.004314673s) [noise=true , address=45.91.116.12:4226 , piece_id=WYJOXZI4TCTVIOZIWKKOJ7LMQKJ2Q56TM6AHG2E6GPJ5EMCTMMUQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.291618069 (85h2m15.014181119s) [noise=true , address=45.91.116.10:4082 , piece_id=DOTFN23XOTU7SJL4QZMAQBD53NHZHKPIODSMAJQQTR2LWKAOZYXA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.306946297 (85h2m15.029509337s) [noise=true , address=45.91.116.5:4299 , piece_id=LYURUS26P5XVG6XY5MK6BHRRP6INAID5MZRTFZ4ONZLC6GC5D2ZQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.311631491 (85h2m15.034194551s) [noise=true , address=77.90.31.14:4661 , piece_id=4TKPGW2FXHZYLCYGFGKA3KRXL3SEEKWRH4MLHCREEX6YA6BBW2UA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.321245706 (85h2m15.043808746s) [noise=true , address=45.91.116.7:4010 , piece_id=X43FRYXKATVJO44TE24ITU4QE2NIIAG3PUDWBNQ4RV4ZWEXWMRIA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.353862883 (85h2m15.076425923s) [noise=true , address=77.90.31.13:4627 , piece_id=MLOE42AP7JESZJDX6QBJQ473MVOML5TULDZJTOB3EIW7YNXSYXMA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.391572863 (85h2m15.114135903s) [noise=true , address=77.90.31.11:4555 , piece_id=ZWELWJLZIAWK4DYIQWZT2FUGZTBTKVJ2QMQGLS4YPG62DYRJFSQQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.395874373 (85h2m15.118437413s) [noise=true , address=77.90.31.15:4698 , piece_id=V5JJIMZDE7PJ4YKMF32EY4BBZ23TXZGSID4RR25732Q4W4VJ256Q, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.398440032 (85h2m15.121003072s) [noise=true , address=45.91.116.14:4190 , piece_id=FP6VAMHTXSCQB3MA4HJ7PK2MS6LBZBX36XR4U36RYJSNCHJHZUKA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:05.407792857 (85h2m15.130355897s) [noise=true , address=45.91.116.13:4154 , piece_id=OFHZPW64RCX6YQNFHGIKLK2JJ3ETNC3A5RGGJMQTC6DSMWWR3UPA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: <nil> Jul 28 00:38:32.090405603 (85h2m41.812968643s) [noise=true , address=104.36.82.14:4223 , piece_id=V5YTIWRKZY2NZJ6AAOFHOO26VE775LP3SL3VAFHJAEK4VQYO6VRA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:42696->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090425331 (85h2m41.812988371s) [noise=true , address=104.36.82.11:4115 , piece_id=SLAUZGYHW44M2HNJ5QVRK3DHQDZGGJ6CWN5LSKAPZSICLJC56BKQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090458645 (85h2m41.813021685s) [noise=true , address=104.36.82.16:4295 , piece_id=CJK6NMD4PFJPGAFKHBIGDLNEDMFCQJA6RTUCP3MV4AXIMXLPABAA, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 188.166.225.170:48192->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090475527 (85h2m41.813038567s) [noise=true , address=104.36.82.12:4151 , piece_id=PLWWMA3ROPRSVB4NJVUEW3TBKRQQ4BJYWHM45BV6QEOPPYHKDEYQ, seg_pos={0 0}, upload=81599]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: write tcp 188.166.225.170:34694->104.36.82.12:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:32.090505685 (85h2m41.813068725s) [seg_pos={0 0}, upload=81599]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:38:32.328034839 (85h2m42.050597879s) [upload=81599]: Done uploading segments... Jul 28 00:38:32.328061009 (85h2m42.050624049s) [upload=81599]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.239614608s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [...........................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231194223s FAIL (2025-07-28 00:38:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== .51:4063 , piece_id=UFOHDUIFDYZ7YIBNQ35MTCXPLRLBHBOWBDZIVFFANTBILK4IXIUA, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:24.266483250 (85h1m33.98904629s) [noise=true , address=104.36.82.39:4759 , piece_id=BGQTQ5MAAG223G3JQMKAWUQ3VFQUBUM4TRNNBUYDTXR5RWXFAQSQ, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:24.266528907 (85h1m33.989091947s) [noise=true , address=104.36.82.10:4082 , piece_id=S4LLMRMLLLGBKZVAOU64YRTTPOBUY7RJCRE2XDFSYKZQOHCUKS2A, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:24.266734811 (85h1m33.989297851s) [noise=true , address=104.36.82.14:4232 , piece_id=SGO45O3YHRKDLJHMYC6MDK5XGMVLVIITKKVUMUZUGLC3SW6PUBRQ, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:24.266938702 (85h1m33.989501742s) [noise=true , address=104.36.82.46:4507 , piece_id=AZBZNPIEUANURICPT5HHNIBJOONXOYSZKMRTCMWCGOIK5MQ6O25Q, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:24.267146859 (85h1m33.989709899s) [noise=true , address=64.49.13.139:4473 , piece_id=ZSD2KX4MOD32MYM2ZDCNAOCRURMIWC2K6RXIY54CATNRSDT7ALFA, seg_pos={0 0}, upload=81593]: Uploading piece... Jul 28 00:37:25.455809245 (85h1m35.178372285s) [upload=81593]: Next returned nil segment Jul 28 00:37:25.455836717 (85h1m35.178399758s) [upload=81593]: Waiting for error group managing segments... Jul 28 00:37:25.760609097 (85h1m35.483172137s) [noise=true , address=64.49.13.139:4473 , piece_id=ZSD2KX4MOD32MYM2ZDCNAOCRURMIWC2K6RXIY54CATNRSDT7ALFA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: <nil> Jul 28 00:38:23.180165089 (85h2m32.902728129s) [noise=true , address=104.36.82.55:4203 , piece_id=UMCK26FYUQZU44KYNT4YUB7H6FLMYEUJRMH2X6CGOP7PQRXPRLAA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12ZewP4Cfpax2814Hqbtfvp4PzcDX455Gv7WwSTSwt1NTMXzYhU, address:104.36.82.55:4203): protocol: write tcp 188.166.225.170:45912->104.36.82.55:4203: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180209904 (85h2m32.902772944s) [noise=true , address=104.36.82.51:4063 , piece_id=UFOHDUIFDYZ7YIBNQ35MTCXPLRLBHBOWBDZIVFFANTBILK4IXIUA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12M7QPxHpz7ENUXau3eW71JSTjCqMkyhY4ueY38qRhUULHtvs6D, address:104.36.82.51:4063): protocol: write tcp 188.166.225.170:40210->104.36.82.51:4063: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180250091 (85h2m32.902813141s) [noise=true , address=104.36.82.10:4082 , piece_id=S4LLMRMLLLGBKZVAOU64YRTTPOBUY7RJCRE2XDFSYKZQOHCUKS2A, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12DKsNCtqxLvpJwLszdLs74V3eb8NH2VAV9V3bngzUczawmNoLK, address:104.36.82.10:4082): protocol: write tcp 188.166.225.170:37834->104.36.82.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180280529 (85h2m32.902843569s) [noise=true , address=104.36.82.14:4232 , piece_id=SGO45O3YHRKDLJHMYC6MDK5XGMVLVIITKKVUMUZUGLC3SW6PUBRQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1jWewg226E4BrBxqcK38n4ZjHx3toQeNdbc5HeU2EoykztXwyX, address:104.36.82.14:4232): protocol: write tcp 188.166.225.170:44004->104.36.82.14:4232: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180311869 (85h2m32.902874909s) [noise=true , address=104.36.82.46:4507 , piece_id=AZBZNPIEUANURICPT5HHNIBJOONXOYSZKMRTCMWCGOIK5MQ6O25Q, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: write tcp 188.166.225.170:43634->104.36.82.46:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180343349 (85h2m32.902906379s) [noise=true , address=104.36.82.11:4145 , piece_id=DZU7RUV4B6AYCO5BDUWWFWAV7YMQOXO5JZXY2WDG5JZ3LDFVQG4A, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12sTUH6Y4HBG8ocrPAuuFJiVrcAm1nhpNhJXLW4HaCMdtYmyukp, address:104.36.82.11:4145): protocol: rpc: dial tcp 104.36.82.11:4145: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180404015 (85h2m32.902967055s) [noise=true , address=104.36.82.41:4767 , piece_id=XU55VNOGLQXZ7UG3X3EKQUMZMQI3VNIDHBRBAM7TWC4N5MM35JGQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12sfaYCUhNCAyQAPu1jac5dRWR35Qqc4ZcKg75mp7nVQak295ug, address:104.36.82.41:4767): protocol: write tcp 188.166.225.170:41144->104.36.82.41:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180438031 (85h2m32.903001071s) [noise=true , address=104.36.82.39:4759 , piece_id=BGQTQ5MAAG223G3JQMKAWUQ3VFQUBUM4TRNNBUYDTXR5RWXFAQSQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12AEfTC9phpBk8uTbN6k7oDvz6svgwdPH1PaU8jfuuBxhHTRtTR, address:104.36.82.39:4759): protocol: rpc: dial tcp 104.36.82.39:4759: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180513074 (85h2m32.903076114s) [noise=true , address=104.36.82.12:4177 , piece_id=XZAOYDSQRO6RD5LLRPCKYKR46XKVHAXUGDEPLV6QGFNHFYHE5RUA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12rFuJ2JyBGWtaASJNnTdHoeKUY1mCfDYfJ74BkxK64U9it8FKF, address:104.36.82.12:4177): protocol: write tcp 188.166.225.170:58916->104.36.82.12:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180563550 (85h2m32.90312659s) [noise=true , address=104.36.82.15:4266 , piece_id=26EINTJPBFTRG7TVE33NCZSEOKSI57UFKNXZSWKXBX4ZGLSBMEBQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1wH8jka9Jw8JLYdGKuit8XoJjVLJRDKJ5ePgeei16eax6mk3j, address:104.36.82.15:4266): protocol: write tcp 188.166.225.170:41424->104.36.82.15:4266: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180604489 (85h2m32.903167529s) [noise=true , address=104.36.82.40:4968 , piece_id=MDCGQ4AV3HB35W3WOZX2LUCGJFRRYPP7PBC3IIISPO45AFEIYFRA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1J98jLqn1SVMYjTYxKnNqYbRxSQ5vaUnq2G7XpVngHTaWRJ8oX, address:104.36.82.40:4968): protocol: rpc: dial tcp 104.36.82.40:4968: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180612875 (85h2m32.903175915s) [noise=true , address=104.36.82.54:4166 , piece_id=5KXZZUWKR5S33GYH6TRAJ5CFXJZE75DRTZYB4NE2DHBLT6RQVKKQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 188.166.225.170:49180->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180647992 (85h2m32.903211032s) [noise=true , address=104.36.82.14:4245 , piece_id=XG37BA54JEOWOQ4WH52EVPXZRXLB67GYLE73J62YQ6B4SWXQPXYQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1Zu8teNPTXn638grHKwtJqz4UEY7oFYLx1DHvuVP6RZXcatVqX, address:104.36.82.14:4245): protocol: write tcp 188.166.225.170:47460->104.36.82.14:4245: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180660877 (85h2m32.903223917s) [noise=true , address=104.36.82.9:4010 , piece_id=7T6JRA7PS7TEB26DFFINBUQ7UB23HHROMBP6ZJG6UNSOTKQGFJOA, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:19XkukzF8ZCLdHLTYQTYpX5iF4tvT1iM6Zxxs1ZNQ4r5X4ousp, address:104.36.82.9:4010): protocol: write tcp 188.166.225.170:42590->104.36.82.9:4010: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180675254 (85h2m32.903238294s) [noise=true , address=104.36.82.53:4140 , piece_id=YHSEUMNTKKAMG7W3O2IULBIPHPPCOKRNBWIGHUOX3B67TGITO4SQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:12Wv1U4bCkzg9Ywx4EKRkWG8tMffjwiVBEZ6tHr8scNHfdcWyrL, address:104.36.82.53:4140): protocol: write tcp 188.166.225.170:58768->104.36.82.53:4140: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180691185 (85h2m32.903254215s) [noise=true , address=104.36.82.20:4455 , piece_id=NKYO3OMVZJH46ICU2S7BZA5AQUNECCQRIAUX75PWQ62L24EFLW3A, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1X8uFZXHVMycvk6tELmYQyWoAS2sSFmNNWhvQFApvoWXuXFcj4, address:104.36.82.20:4455): protocol: write tcp 188.166.225.170:35466->104.36.82.20:4455: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180717195 (85h2m32.903280225s) [noise=true , address=104.36.82.52:4094 , piece_id=BJTLX2AB5U5Z5QSM7FTGVOPEFYCGOOEJBGAK3OTXHAQQYU4YM2SQ, seg_pos={0 0}, upload=81593]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 188.166.225.170:44258->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:38:23.180745699 (85h2m32.903308729s) [seg_pos={0 0}, upload=81593]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:38:23.409718713 (85h2m33.132281753s) [upload=81593]: Done uploading segments... Jul 28 00:38:23.409746346 (85h2m33.132309386s) [upload=81593]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.231206426s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [...........................X.......X.XX....X....X.......X...] duration: 5.816133798s FAIL (2025-07-28 00:38:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.816153004s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [...........X............X.......................X...........] duration: 1.84981672s FAIL (2025-07-28 00:38:10 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.849834975s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [............................X...XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.231662629s FAIL (2025-07-28 00:37:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== =104.36.82.45:4940 , piece_id=Z6AU7C5OD2QWEFEAEJZ244J4UKK7FG7Y6VT7PENN2YXLYJ3YZM2Q, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.730719933 (85h0m33.453282993s) [noise=true , address=104.36.82.15:4262 , piece_id=GZNEFZNVOWSGPJ4FEERKNCT7HRVO4YGNRO3LVAVU5UL2BSL3RS3Q, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.730859911 (85h0m33.453422941s) [noise=true , address=104.36.82.14:4229 , piece_id=IR2CMHPIWPOJRKACLHUKR6WODY2QYUKF5U2D6MXKMMRM66QECWZQ, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.731137944 (85h0m33.453700984s) [noise=true , address=104.36.82.5:4842 , piece_id=KOAK7K7CIZOKNB7BEB6KP5EJJO4RYFQDQSJD3IESBQY6UU3TFO6Q, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.731386289 (85h0m33.453949329s) [noise=true , address=104.36.82.54:4157 , piece_id=5JDBZYOG4B446WAE77YNEUNRF2YZRWPHN4QJP5665ATV5S5N7QZA, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.731626378 (85h0m33.454189418s) [noise=true , address=104.36.82.43:4863 , piece_id=ZIKPNZPR5I6A6WB5SX62525MUDJGVALO5KOWMUCSDAUME5NEBN2A, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.731692855 (85h0m33.454255895s) [noise=true , address=79.127.201.215:4355 , piece_id=4J22OMFGTJORJOBN56GPGTJZ7YFPOXLH4YYWRFJLEQELBEHFUCGA, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.731927033 (85h0m33.454490073s) [noise=true , address=104.36.82.18:4399 , piece_id=OD6UMBRKY4JYMODZ2HVKUHZXZJZNQT3CGUS5PVMKSIMZKSCNHDCQ, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:23.732148737 (85h0m33.454711777s) [noise=true , address=104.36.82.53:4155 , piece_id=EOU25V4X4JQOLCO3HH4YY62R2ZA5FV3JLEZZAOFAN6XRUVLG6E4Q, seg_pos={0 0}, upload=81576]: Uploading piece... Jul 28 00:36:24.892142421 (85h0m34.614705461s) [upload=81576]: Next returned nil segment Jul 28 00:36:24.892164061 (85h0m34.614727101s) [upload=81576]: Waiting for error group managing segments... Jul 28 00:36:25.189753289 (85h0m34.912316339s) [noise=true , address=64.49.13.154:4932 , piece_id=M73RZ3CUUMCODQI2GLDPXRYQPXSGXNZNSUUQ3276DCQDAECJQ7FQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: <nil> Jul 28 00:36:29.598541183 (85h0m39.321104223s) [noise=true , address=79.127.201.215:4355 , piece_id=4J22OMFGTJORJOBN56GPGTJZ7YFPOXLH4YYWRFJLEQELBEHFUCGA, seg_pos={0 0}, upload=81576]: Done uploading piece. err: <nil> Jul 28 00:37:22.639412287 (85h1m32.361975327s) [noise=true , address=104.36.82.15:4262 , piece_id=GZNEFZNVOWSGPJ4FEERKNCT7HRVO4YGNRO3LVAVU5UL2BSL3RS3Q, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:12nPw17QDTyrgDBBNKuunFWuDqPV5Rd2GpM7Loc6Ws9vAAZz22s, address:104.36.82.15:4262): protocol: rpc: dial tcp 104.36.82.15:4262: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639480808 (85h1m32.362043848s) [noise=true , address=104.36.82.12:4168 , piece_id=W6UT7MIYXRARWEVAYBGCL55LX7VTLHOESWUJLNGOCTPQXD6OZ74A, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:12XJ5kTKY8pLq3uDJZfFVkTgS56Xx8pG2R5i2A9VRNDEckdKnTF, address:104.36.82.12:4168): protocol: rpc: dial tcp 104.36.82.12:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639535122 (85h1m32.362098162s) [noise=true , address=104.36.82.42:4830 , piece_id=6DHOI5O3TYJLPCMCLIHTY7JYQFP6PZB7GBHAWDTHR27FV33MTHWQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639586661 (85h1m32.362149701s) [noise=true , address=104.36.82.13:4210 , piece_id=LKOFXIAU4YNQALXHRJM55JJI2P4NJ6Y4LOQXKMWKQEVVX4LXHYJQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1ybrcx8SPgyEsJWq2P1KYaTwTj5v5BfENKNEZZxaKRvKvuqTiL, address:104.36.82.13:4210): protocol: rpc: dial tcp 104.36.82.13:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639635715 (85h1m32.362198755s) [noise=true , address=104.36.82.43:4863 , piece_id=ZIKPNZPR5I6A6WB5SX62525MUDJGVALO5KOWMUCSDAUME5NEBN2A, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1hoK1e9x7ms16dWqXpNpfHA97U74iX9DkT7n73YUTcePgbMatZ, address:104.36.82.43:4863): protocol: rpc: dial tcp 104.36.82.43:4863: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639690119 (85h1m32.362253159s) [noise=true , address=104.36.82.18:4399 , piece_id=OD6UMBRKY4JYMODZ2HVKUHZXZJZNQT3CGUS5PVMKSIMZKSCNHDCQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1QPwXaFGpwMLXMiSnkE1eycqZkUSnDYMj9sFYZ2BPrPbenw9uK, address:104.36.82.18:4399): protocol: rpc: dial tcp 104.36.82.18:4399: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639739843 (85h1m32.362302883s) [noise=true , address=104.36.82.53:4155 , piece_id=EOU25V4X4JQOLCO3HH4YY62R2ZA5FV3JLEZZAOFAN6XRUVLG6E4Q, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:17wRBzmgMvpNyXy9SZEwEjwc4W5TfKzTj8vaa9c2HXnPNLMMYi, address:104.36.82.53:4155): protocol: rpc: dial tcp 104.36.82.53:4155: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639808355 (85h1m32.362371395s) [noise=true , address=104.36.82.14:4229 , piece_id=IR2CMHPIWPOJRKACLHUKR6WODY2QYUKF5U2D6MXKMMRM66QECWZQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:12uiCMvJ7pnwXfN57CVu8zR73tikwUZiXr3sG7Xotd9w6KVnViL, address:104.36.82.14:4229): protocol: write tcp 188.166.225.170:37942->104.36.82.14:4229: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639827492 (85h1m32.362390532s) [noise=true , address=104.36.82.20:4442 , piece_id=CXAQEIOMH26XTHKGPRGFTSFHJAJF4NE6AV7W7GWC3JTQP4347AMQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:125KyoqeSzbjP8XPhXqw2AenTDiLtVmt5KpLBD7ZW7YC9e9zJu9, address:104.36.82.20:4442): protocol: rpc: dial tcp 104.36.82.20:4442: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639867628 (85h1m32.362430678s) [noise=true , address=104.36.82.54:4157 , piece_id=5JDBZYOG4B446WAE77YNEUNRF2YZRWPHN4QJP5665ATV5S5N7QZA, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639886304 (85h1m32.362449354s) [noise=true , address=104.36.82.41:4776 , piece_id=SC4PNYVR3F3HT5XD7L76TUFMSKHP6WLVZAFL4PLHYDLZ67HHSZMA, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1Hjw4yBSJfpaGqUWqAqfqc1P9ThM6kwd5vMgbLCrSR8z9eiqH7, address:104.36.82.41:4776): protocol: write tcp 188.166.225.170:36708->104.36.82.41:4776: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639919868 (85h1m32.362482918s) [noise=true , address=104.36.82.5:4842 , piece_id=KOAK7K7CIZOKNB7BEB6KP5EJJO4RYFQDQSJD3IESBQY6UU3TFO6Q, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:12q4VdT5BcjUb16PBYHK6zrnW8dqENKrcGx9zdLtNefDVB1MUSn, address:104.36.82.5:4842): protocol: rpc: dial tcp 104.36.82.5:4842: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639959083 (85h1m32.362522123s) [noise=true , address=104.36.82.56:4261 , piece_id=FPHK2HCZQJEYWC552A6R4TW37QXCKZ2ZUZJIGWCE7WRIM42CLFNQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:12YNKaVEsXzHe6XGQuv6wR1qqbv6dbzg1xT5Aqkew2ZV3z3tMMf, address:104.36.82.56:4261): protocol: rpc: dial tcp 104.36.82.56:4261: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.639972098 (85h1m32.362535138s) [noise=true , address=104.36.82.10:4107 , piece_id=APQSGUREN5MSCDKLOH7EH72AFLCDSPZRINWJXUXM6XQV4NLFQTQQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1UeLnfo3bu6nCXkQDATe2yNhQEn6kZXicg6CFU9mGB37c1QJk4, address:104.36.82.10:4107): protocol: write tcp 188.166.225.170:36482->104.36.82.10:4107: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.640013266 (85h1m32.362576306s) [noise=true , address=104.36.82.45:4940 , piece_id=Z6AU7C5OD2QWEFEAEJZ244J4UKK7FG7Y6VT7PENN2YXLYJ3YZM2Q, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr, address:104.36.82.45:4940): protocol: rpc: dial tcp 104.36.82.45:4940: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.640041711 (85h1m32.362604751s) [noise=true , address=104.36.82.22:4530 , piece_id=EFIMSLM3BTEF3ZELMANKFT3USJRMYEDAIKQ7CFDAD2YXD4JL3IKQ, seg_pos={0 0}, upload=81576]: Done uploading piece. err: ecclient: upload failed (node:1akHUSoQFafzkGTJ385iYaowc87pJBDNfU5hodSCUDFa5ojN3v, address:104.36.82.22:4530): protocol: write tcp 188.166.225.170:56556->104.36.82.22:4530: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:37:22.640089823 (85h1m32.362652853s) [seg_pos={0 0}, upload=81576]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:37:22.868604711 (85h1m32.591167751s) [upload=81576]: Done uploading segments... Jul 28 00:37:22.868631402 (85h1m32.591194442s) [upload=81576]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.231675755s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........................X.X..........X.....X.X...........] duration: 5.972317602s FAIL (2025-07-28 00:37:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.972343611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X.......X.XX....X....X.......X..] duration: 5.763701308s FAIL (2025-07-28 00:37:05 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.763729692s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [............X............X.......................X..........] duration: 1.857680335s FAIL (2025-07-28 00:36:56 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.857709732s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X.......X.XX....X....X.......X.] duration: 5.831861438s FAIL (2025-07-28 00:36:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.831888931s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 28 out of 60 is failed (threshold=3) [.............................X...XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.234720931s FAIL (2025-07-28 00:36:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 1) ======LOGS====== piece... Jul 28 00:35:23.269352912 (84h59m32.991915952s) [noise=true , address=104.36.82.15:4277 , piece_id=6LWJIDNO5URKOPXM3JLQA4EXLOK67OZJTFLTYMXTMVBZZYGYQZDQ, seg_pos={0 0}, upload=81562]: Uploading piece... Jul 28 00:35:23.269466118 (84h59m32.992029158s) [noise=true , address=104.36.82.14:4230 , piece_id=ABQ7QRVKKCPXQWSCQPYHJMZ3KY4ULO2DONZH36HHJTTGZZJWYHCQ, seg_pos={0 0}, upload=81562]: Uploading piece... Jul 28 00:35:23.269612368 (84h59m32.992175398s) [noise=true , address=104.36.82.22:4534 , piece_id=5RT7VVHISZAINBGDXTAOLBCWGGDTZWAU2PRPKJSW5AQGLRZG5W2Q, seg_pos={0 0}, upload=81562]: Uploading piece... Jul 28 00:35:23.269729132 (84h59m32.992292162s) [noise=true , address=104.36.82.55:4212 , piece_id=ECF7CNXJXNVHQM4V6GOTIHNUBH6GW3MNBT5B54Q4EALKHYH7TAJQ, seg_pos={0 0}, upload=81562]: Uploading piece... Jul 28 00:35:23.270026290 (84h59m32.99258932s) [noise=true , address=104.36.82.4:4823 , piece_id=27LYFLYI25EQWWT3AR3IF6V62ETPUL76VHC6R2UNNAMUJ27HVRWQ, seg_pos={0 0}, upload=81562]: Uploading piece... Jul 28 00:35:24.786129718 (84h59m34.508692758s) [upload=81562]: Next returned nil segment Jul 28 00:35:24.786147271 (84h59m34.508710302s) [upload=81562]: Waiting for error group managing segments... Jul 28 00:35:28.159006254 (84h59m37.881569304s) [noise=true , address=79.127.201.215:4358 , piece_id=WCQGOHD4IBEY62YJR3UAEOXMRLU4AEKOIKZKWXRCBZA44DOAEHIQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: <nil> Jul 28 00:36:22.168242852 (85h0m31.890805892s) [noise=true , address=104.36.82.10:4110 , piece_id=B4RFMNE4W27RCXIM76MSZWLOQF4BEYG4THZCR5CULPSFYC3O4CTA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1ADbnPJGRTw2NqRzhy7ptojGYoNdJgqmjuWBSjcrQK53Wm7LXi, address:104.36.82.10:4110): protocol: write tcp 188.166.225.170:58348->104.36.82.10:4110: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168289011 (85h0m31.890852041s) [noise=true , address=104.36.82.56:4247 , piece_id=3DVXN6QMQRORBODD623GKFY53K53HWDYR5GWTEJONDGF7GRTGJSQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:12PzKreuPT9X594ZTTYbXanNdaFVVJsnKL99fumjXymnRa9jeEq, address:104.36.82.56:4247): protocol: write tcp 188.166.225.170:60228->104.36.82.56:4247: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168325089 (85h0m31.890888119s) [noise=true , address=104.36.82.5:4865 , piece_id=LEA6LQVCH42FAIWSDDCLUBZL4XNXBHVUKEAWKALHRHYXDVLB2YXA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1kqNGahsBw6VsV3JTALvPyGshuSFaAdRSo94cQnCXF3ak54KtL, address:104.36.82.5:4865): protocol: write tcp 188.166.225.170:58972->104.36.82.5:4865: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168329919 (85h0m31.890892959s) [noise=true , address=104.36.82.4:4823 , piece_id=27LYFLYI25EQWWT3AR3IF6V62ETPUL76VHC6R2UNNAMUJ27HVRWQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1JWhB5QxbZsTD863tEfkEN1CUbKBMQvMJ9ZaiMAwKvxxXyDkEq, address:104.36.82.4:4823): protocol: write tcp 188.166.225.170:52362->104.36.82.4:4823: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168382429 (85h0m31.890945469s) [noise=true , address=104.36.82.44:4896 , piece_id=MUFEHJUJRAJOTSXVDII2EWE64SF7ETU2LFMQCQCX54GHLELKCZBA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:12ndSgjgfZMVQni5tw1ugdTrhJt4c4NagdvgYTbSbjatYYtfu1o, address:104.36.82.44:4896): protocol: write tcp 188.166.225.170:49016->104.36.82.44:4896: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168411515 (85h0m31.890974555s) [noise=true , address=104.36.82.3:4795 , piece_id=UOX7MD5SFADWST5GXA7P43PRH72KFAQ6WH43JSYLQ6HV52JUA7AA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1rL9dLWui62A3vRgBSNw7QQf7g2uubwW85to4ioCrhyao2Pupo, address:104.36.82.3:4795): protocol: write tcp 188.166.225.170:37886->104.36.82.3:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168419149 (85h0m31.890982189s) [noise=true , address=104.36.82.21:4501 , piece_id=OOSVRQJLBBIU7VLWIYUPEAJTVG6UYVVPFLGB4MJC55XDCYN73UBA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1e1R3gjh5CYJdDqck7xcR6A19Hb443FWbX8X2AQikxt5f4AVvU, address:104.36.82.21:4501): protocol: write tcp 188.166.225.170:46752->104.36.82.21:4501: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168451953 (85h0m31.891014993s) [noise=true , address=104.36.82.54:4164 , piece_id=LAUJBQOHIUHQMNR6ORTBSNQWRDRAUZZQO5D6ZWLPVIQ677T2DNTQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1gUXWffNccBoXWftiMWHHSTeEpA5zrd2cHz4uXX5j75GeEK9s2, address:104.36.82.54:4164): protocol: write tcp 188.166.225.170:33612->104.36.82.54:4164: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168479215 (85h0m31.891042255s) [noise=true , address=104.36.82.5:4861 , piece_id=IQSNXYRYOB5MPF75DGVYDND2UW27FGFEFWZCF33RHZ4XGBLSN6KA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1SqXobRvE3NHfzYxdJ8sedJq3W9pQ6KpMLMiojUhpsCigeeNuS, address:104.36.82.5:4861): protocol: write tcp 188.166.225.170:57942->104.36.82.5:4861: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168501897 (85h0m31.891064937s) [noise=true , address=104.36.82.53:4142 , piece_id=P2AL43Z2P4M252YI4IZRKPMZ6SJKPAD2MOSYDT72PMBNT7SW4WEA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1yLPWJs1RfZgFisHUpW3wwiFsvV98yz7adKtAKAK3unq1KZCSK, address:104.36.82.53:4142): protocol: write tcp 188.166.225.170:43384->104.36.82.53:4142: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168517698 (85h0m31.891080738s) [noise=true , address=104.36.82.14:4230 , piece_id=ABQ7QRVKKCPXQWSCQPYHJMZ3KY4ULO2DONZH36HHJTTGZZJWYHCQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:15hXnJMgf7FAJJh5y1S35eNLdvvqQR4yFot9M8FAmwJDyYVsWf, address:104.36.82.14:4230): protocol: write tcp 188.166.225.170:34776->104.36.82.14:4230: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168537256 (85h0m31.891100296s) [noise=true , address=104.36.82.22:4534 , piece_id=5RT7VVHISZAINBGDXTAOLBCWGGDTZWAU2PRPKJSW5AQGLRZG5W2Q, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1Y7PoZ6DqyK6p456p8gQW3DRcK3g5Y6s3i55XoQwcMuvgBFd6t, address:104.36.82.22:4534): protocol: write tcp 188.166.225.170:41252->104.36.82.22:4534: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168594124 (85h0m31.891157164s) [noise=true , address=104.36.82.13:4204 , piece_id=ST3LCMJEH4N3Q2LFE5ZR7EHFCBHLA6BALZQLKDCRHCYJYOOPFM3A, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:12R53RRLYuJc7r9Z1W9n6guzrr5wF4T4vuGMEzjWAPBJwSCJq1g, address:104.36.82.13:4204): protocol: write tcp 188.166.225.170:35608->104.36.82.13:4204: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168614353 (85h0m31.891177393s) [noise=true , address=104.36.82.49:4011 , piece_id=NL27FJLCGGPHGV2G57FTQXRE6NW3DIRKSTEJKCPT7W33GOWLQALQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1DV1WUL3dGBjH23a6xtetmqmLE9fER1FQf53gmYFxZiP25ra8N, address:104.36.82.49:4011): protocol: write tcp 188.166.225.170:51142->104.36.82.49:4011: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168623210 (85h0m31.89118625s) [noise=true , address=104.36.82.55:4212 , piece_id=ECF7CNXJXNVHQM4V6GOTIHNUBH6GW3MNBT5B54Q4EALKHYH7TAJQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: write tcp 188.166.225.170:60196->104.36.82.55:4212: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168633139 (85h0m31.891196179s) [noise=true , address=104.36.82.20:4457 , piece_id=SSUFWEWEJTWDCY2RGFOF3JUSKQFSLGKOQCDI4GDSP2VNCOFERCTA, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:16sDrSF9mwW9cQ2h8N5TfUKrnwPAwtEfed6AwCEAT95r5iY9vQ, address:104.36.82.20:4457): protocol: write tcp 188.166.225.170:60762->104.36.82.20:4457: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168640363 (85h0m31.891203393s) [noise=true , address=104.36.82.15:4277 , piece_id=6LWJIDNO5URKOPXM3JLQA4EXLOK67OZJTFLTYMXTMVBZZYGYQZDQ, seg_pos={0 0}, upload=81562]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: write tcp 188.166.225.170:60122->104.36.82.15:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:36:22.168654239 (85h0m31.891217279s) [seg_pos={0 0}, upload=81562]: Done waiting for segment. successful: 1 optimal: 14 errs: <nil> Jul 28 00:36:22.401018780 (85h0m32.12358182s) [upload=81562]: Done uploading segments... Jul 28 00:36:22.401044589 (85h0m32.123607619s) [upload=81562]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 1) Duration: 1m0.23474114s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............................X.X..........X.....X.X..........] duration: 5.894793817s FAIL (2025-07-28 00:36:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.89482165s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.............X............X.......................X.........] duration: 1.850946732s FAIL (2025-07-28 00:36:02 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.850975646s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..............................X.......X.XX....X....X.......X] duration: 30.255506767s FAIL (2025-07-28 00:35:58 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.255528738s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............................X.X..........X.....X.X.........] duration: 5.767161099s FAIL (2025-07-28 00:35:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767188652s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [..............................X...XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.23433276s FAIL (2025-07-28 00:35:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 2) ======LOGS====== , piece_id=RPHSPO2JQMKG6EEL2G4FSCMEESCMQGHJWNMQH6PW5XYELR6PABMQ, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.186646093 (84h58m20.909209133s) [noise=true , address=64.49.13.154:4932 , piece_id=ILL43ANVYOBRISC3QMBR6JTCF67DMQG6HC2CNJFJMIPCBTB7KLAQ, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.187015289 (84h58m20.909578339s) [noise=true , address=104.36.82.6:4894 , piece_id=AUZEUHAM2XVKRRK5AGS35JKSHVXCPYBYPVB53VZ2UFKPPRHMQHIA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.187865666 (84h58m20.910428706s) [noise=true , address=104.36.82.15:4263 , piece_id=XWWC3B4EX5CE7Q2S5QCJAQJRGQNT4KICA56QMMYJHHPNSQVDQZ7Q, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.188578550 (84h58m20.91114159s) [noise=true , address=104.36.82.14:4242 , piece_id=6XHODN4FXR4L7KDIVRI7XZ43JSQBD6E26HWR5DXS6XBTNWPNKUFA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.193602393 (84h58m20.916165433s) [noise=true , address=104.36.82.3:4783 , piece_id=C3E35WFHLV6QATOFQNFEEH3CCITV35TOUOKTQ5BWSVQLKWXKHQDA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:11.194456497 (84h58m20.917019537s) [noise=true , address=104.36.82.51:4059 , piece_id=WAAFHIH5XM6WT7N3WLZ3YLTOMPILZADKLJQYZEWLHX2ENRJ5VCMA, seg_pos={0 0}, upload=81541]: Uploading piece... Jul 28 00:34:12.331374429 (84h58m22.053937469s) [upload=81541]: Next returned nil segment Jul 28 00:34:12.331391842 (84h58m22.053954883s) [upload=81541]: Waiting for error group managing segments... Jul 28 00:34:12.640722704 (84h58m22.363285744s) [noise=true , address=64.49.13.154:4932 , piece_id=ILL43ANVYOBRISC3QMBR6JTCF67DMQG6HC2CNJFJMIPCBTB7KLAQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: <nil> Jul 28 00:34:13.747173143 (84h58m23.469736183s) [noise=true , address=77.90.31.16:4748 , piece_id=5PMV6PPODQQENZMCG7OXUKI7Q36XJI7JDX6D5B2ULEQWG5XJJE6A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: <nil> Jul 28 00:35:10.081128191 (84h59m19.803691241s) [noise=true , address=104.36.82.35:4615 , piece_id=PDBNLHVRWOGD6VQRYUXRARQFJKXBUANFL6YAUAJHXLFRPBNO23LA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12NJ5w8pAWCudDA9Xs4ES7b2evnbscesd682wJDh2gMMZ3Jf2zc, address:104.36.82.35:4615): protocol: write tcp 188.166.225.170:50616->104.36.82.35:4615: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081179818 (84h59m19.803742859s) [noise=true , address=104.36.82.13:4193 , piece_id=RPHSPO2JQMKG6EEL2G4FSCMEESCMQGHJWNMQH6PW5XYELR6PABMQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:15XNLgptmXprcfkVMJsuzY2WDyHiZDpLetiBpmjXhb4c9q7yMd, address:104.36.82.13:4193): protocol: write tcp 188.166.225.170:60864->104.36.82.13:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081214205 (84h59m19.803777235s) [noise=true , address=104.36.82.6:4894 , piece_id=AUZEUHAM2XVKRRK5AGS35JKSHVXCPYBYPVB53VZ2UFKPPRHMQHIA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12fZsj77XZ8CnYD6iwzEVKPn6yLyxWMtibzsRXcuGKwNxEXB6Hq, address:104.36.82.6:4894): protocol: write tcp 188.166.225.170:38976->104.36.82.6:4894: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081249612 (84h59m19.803812653s) [noise=true , address=104.36.82.53:4154 , piece_id=LMMA7JHS2NGKKX3OB3MNWYCM3UPNGPQLXS347PCXYDLLPBNZWOOQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1vbGunXCRMkmfvBqHPmbuW7fBLKck4Vh9MGsX1q7P6VfX6DKqB, address:104.36.82.53:4154): protocol: write tcp 188.166.225.170:52900->104.36.82.53:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081281223 (84h59m19.803844253s) [noise=true , address=104.36.82.12:4154 , piece_id=GQYNCE7AITKLZ7DOHQNZ7LYKEXAJVI2FMUT6IBXCJPIL3UGFM36A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: write tcp 188.166.225.170:47294->104.36.82.12:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081313064 (84h59m19.803876094s) [noise=true , address=104.36.82.55:4193 , piece_id=HYOCXSRM47ZMO7QUU35JTPU3GVLGSF6TWWEB3ZEUUD3YHFFO5VVA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8, address:104.36.82.55:4193): protocol: rpc: dial tcp 104.36.82.55:4193: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081347079 (84h59m19.803910109s) [noise=true , address=104.36.82.14:4242 , piece_id=6XHODN4FXR4L7KDIVRI7XZ43JSQBD6E26HWR5DXS6XBTNWPNKUFA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1inR8rjUp4fRQWcaD2S7MRxveHyFxKoutWoEiuKQG3C895PaMz, address:104.36.82.14:4242): protocol: write tcp 188.166.225.170:59762->104.36.82.14:4242: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081382457 (84h59m19.803945487s) [noise=true , address=104.36.82.51:4059 , piece_id=WAAFHIH5XM6WT7N3WLZ3YLTOMPILZADKLJQYZEWLHX2ENRJ5VCMA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12fQsGGr5K6TVMU7XC2v1mQBFwHznj5ugYvYLmmAUnP81AyHAjY, address:104.36.82.51:4059): protocol: write tcp 188.166.225.170:33624->104.36.82.51:4059: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081416692 (84h59m19.803979732s) [noise=true , address=104.36.82.54:4191 , piece_id=FJXF54KAMXXFSJPM5DGH56RTPCSLTGMFEBBJRTK3FLK4UIF4VKDQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12d7ChCRcN4pJjJ51aSAPitg3zTRxr5eVbRXVZcvbi6t1gifWmw, address:104.36.82.54:4191): protocol: rpc: dial tcp 104.36.82.54:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081457481 (84h59m19.804020521s) [noise=true , address=104.36.82.3:4783 , piece_id=C3E35WFHLV6QATOFQNFEEH3CCITV35TOUOKTQ5BWSVQLKWXKHQDA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12ito72D7vikRdjK7BPt4LnhAK84bzi8AbZZPNWrTqmbjwQT1N, address:104.36.82.3:4783): protocol: write tcp 188.166.225.170:42850->104.36.82.3:4783: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081493920 (84h59m19.80405696s) [noise=true , address=104.36.82.52:4090 , piece_id=73SMTCFCCI76TX3TG2SYNSDPTBS5NOCKEG5HAVKKB7DKNSY464ZA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: write tcp 188.166.225.170:42426->104.36.82.52:4090: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081533115 (84h59m19.804096155s) [noise=true , address=104.36.82.12:4169 , piece_id=443NIAZIZBAABGE64P7H5RC3J4VN3EBYEVOLDV6UIZFDR3NOSJIA, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:12AcJsPCQkjCgLxMTVcjMC5BpEe9RqzcC4CNC3oJDUt4f8cTbNX, address:104.36.82.12:4169): protocol: write tcp 188.166.225.170:35794->104.36.82.12:4169: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081550899 (84h59m19.804113939s) [noise=true , address=104.36.82.13:4220 , piece_id=WPBF76CZLFTULBAUXCRUKY7NUB65GXFWIDD3YF6ADASPSD723E6A, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1AjQmw3H58hTwr2389cmMcwy1gBavBrZB8AWB1gqfoFRUkCpH6, address:104.36.82.13:4220): protocol: write tcp 188.166.225.170:33048->104.36.82.13:4220: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081578412 (84h59m19.804141442s) [noise=true , address=104.36.82.42:4818 , piece_id=T4NRYW7NX2KKQTCXRWAGUJOY354GLZEMKHHYEABPSJZ2TXNP47IQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:122m1dk7cCwZfUKeq4BChF1UjXSt7swvt1L3HdiSybavXhLEcsb, address:104.36.82.42:4818): protocol: write tcp 188.166.225.170:55632->104.36.82.42:4818: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081616073 (84h59m19.804179113s) [noise=true , address=104.36.82.56:4233 , piece_id=3VTZ3QJO5AYNKPGJX34ZN6QGJFXGMDO4J7MPJ4K3ISOK3PS3ARHQ, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: write tcp 188.166.225.170:46066->104.36.82.56:4233: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081644908 (84h59m19.804207948s) [noise=true , address=104.36.82.15:4263 , piece_id=XWWC3B4EX5CE7Q2S5QCJAQJRGQNT4KICA56QMMYJHHPNSQVDQZ7Q, seg_pos={0 0}, upload=81541]: Done uploading piece. err: ecclient: upload failed (node:1f97yBEvcrYHqMq7iovVoju77jzhUYgoQCsUGULeTagugFXrCu, address:104.36.82.15:4263): protocol: write tcp 188.166.225.170:45518->104.36.82.15:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:35:10.081692550 (84h59m19.80425559s) [seg_pos={0 0}, upload=81541]: Done waiting for segment. successful: 2 optimal: 14 errs: <nil> Jul 28 00:35:10.313368144 (84h59m20.035931184s) [upload=81541]: Done uploading segments... Jul 28 00:35:10.313395426 (84h59m20.035958466s) [upload=81541]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 2) Duration: 1m0.234345685s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [..............X............X.......................X........] duration: 1.962413981s FAIL (2025-07-28 00:35:04 UTC) orbiter warning: digest mismatch: after f108761aa7d0920bb363621c2197fd3578093c870514492788074b7848c554e0 != before 8e770a63b4165ee9ca5f7101d5bcd1d72eb0c5d764fc243e1413ffe36e2d1ae6 Duration: 1.962447054s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............................X.X..........X.....X.X........] duration: 5.841326255s FAIL (2025-07-28 00:34:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== : piecestore close: conn closed: op Jul 28 00:28:35.518933456 (84h52m45.241496486s) [noise=true , address=77.90.31.14:4663 , piece_id=D4WW6RGHTKEAO2T5WKRIE7WFK2LSOH3A2KYXPRQROSH2SHCK5LWA, seg_pos={0 0}, upload=81449]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Jul 28 00:28:35.518966329 (84h52m45.241529369s) [seg_pos={0 0}, upload=81449]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 28 00:28:35.519050750 (84h52m45.24161379s) [upload=81449]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 00:28:35.519055230 (84h52m45.24161827s) [upload=81449]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 00:28:35.809458868 (84h52m45.532021898s) [upload=81449]: Done uploading segments... Jul 28 00:28:35.809476372 (84h52m45.532039412s) [upload=81449]: Upload finished. err: <nil> Jul 28 00:29:02.559852893 (84h53m12.282415933s) [upload=81457]: Starting upload Jul 28 00:29:02.559924630 (84h53m12.282487671s) [upload=81457]: Done starting upload Jul 28 00:29:02.560098182 (84h53m12.282661222s) [upload=81457]: Uploading segments... Jul 28 00:29:02.560181612 (84h53m12.282744652s) [upload=81457]: Got next segment. Inline: false Jul 28 00:29:02.560221077 (84h53m12.282784117s) [upload=81457]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 00:29:02.560225937 (84h53m12.282788977s) [upload=81457]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 00:29:03.009059749 (84h53m12.731622819s) [seg_pos={0 0}, upload=81457]: Begin upload segment... Jul 28 00:29:03.009158979 (84h53m12.73172206s) [seg_pos={0 0}, upload=81457]: Done begin upload segment. Jul 28 00:29:03.009178718 (84h53m12.731741788s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009188606 (84h53m12.731751656s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009792902 (84h53m12.732355942s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.009842818 (84h53m12.732405858s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010397319 (84h53m12.732960359s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010459187 (84h53m12.733022227s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010730286 (84h53m12.733293326s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.010957360 (84h53m12.7335204s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011154638 (84h53m12.733717688s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011401190 (84h53m12.73396425s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.011975508 (84h53m12.734538568s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.012958940 (84h53m12.73552198s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013411306 (84h53m12.735974346s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.013856397 (84h53m12.736419437s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.015192333 (84h53m12.737755373s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:03.016283781 (84h53m12.738846841s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Uploading piece... Jul 28 00:29:04.783335657 (84h53m14.505898687s) [upload=81457]: Next returned nil segment Jul 28 00:29:04.783353261 (84h53m14.505916301s) [upload=81457]: Waiting for error group managing segments... Jul 28 00:29:05.060157571 (84h53m14.782720611s) [noise=true , address=45.91.116.12:4226 , piece_id=IVJCX4DRW5MNEPRUJ3MZZI5FVH5VPN2MRMX4242LF5EXADY642GQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.099038051 (84h53m14.821601091s) [noise=true , address=77.90.31.14:4662 , piece_id=M3UZIFESWBE7ICRMSUG6AHQP4OKB4IB2CXIJASIATTWL3HAR5XCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.113329355 (84h53m14.835892395s) [noise=true , address=45.91.116.6:4974 , piece_id=JRKWVG4KHMZPP36IWHCZNS7RRRA3QA3KPGE2PP47BTHMIHBXBLEQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.116510420 (84h53m14.83907345s) [noise=true , address=45.91.116.14:4191 , piece_id=WVDBAGF7VY5GUF6JEOYBNJCQCMNNS2XXIQ3LBYY4LJ3BZ2J2SD7Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131296190 (84h53m14.85385923s) [noise=true , address=77.90.31.10:4520 , piece_id=7KTSJ5NMZCF6HKXVWBON4E6ADIE4SZYH6YJRRF5E66OG3ZZ6PPDQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.131761601 (84h53m14.854324641s) [noise=true , address=77.90.31.12:4590 , piece_id=AXYXD45UZXCL6DUJFUSQWDGK7YL5TNX2L75PCQMLD4TOB7GIETBA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.135486167 (84h53m14.858049197s) [noise=true , address=45.91.116.8:4045 , piece_id=DJQ6G5N6QOF23KJWJ5QG4VQULDK5FMHHW2YOW4CNN5WGGA23BJCQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.137908521 (84h53m14.860471561s) [noise=true , address=45.91.116.10:4082 , piece_id=ET7H6GHFO7TUCBHG2ZJGENBBTJXFACXJCZYEVM666GPSZHWQAL2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.141090389 (84h53m14.863653429s) [noise=true , address=77.90.31.12:4592 , piece_id=VVAGFRV7K23FE2IEUANVG2CXEMB3UPHO7QF6HVY4W7HEDUUXXDYA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.146438030 (84h53m14.869001071s) [noise=true , address=45.91.116.9:4263 , piece_id=KZ6CUSPAQF32EE5V7IBNZPXHZVYUI6E7JUIILPTTB5XQD4IHYO2Q, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.210337970 (84h53m14.93290101s) [noise=true , address=45.91.116.13:4155 , piece_id=OU4L4PVBLIIUJ3622P3YJVA2XBJOGB3IFTW7FQ66BUJXGXW7ALRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:05.271828060 (84h53m14.9943911s) [noise=true , address=77.90.31.18:4806 , piece_id=EDYSRWSRO227H4NZPNGBWFCC7XETV4MCNDVY4MBWH4UFUPAJ7QJA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: <nil> Jul 28 00:29:31.908491111 (84h53m41.631054151s) [noise=true , address=104.36.82.11:4113 , piece_id=D5YGTYWGJOOT3WNJY2MEOVKL25ONCOSTI7KDJLXVDBOAHZLAQBKQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908513103 (84h53m41.631076144s) [noise=true , address=104.36.82.15:4258 , piece_id=RNAF33EEEK4KICEEW66ZUGFVRKEL2RAW3EH3ARKXT3PDO5IEZULA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908534383 (84h53m41.631097423s) [noise=true , address=104.36.82.7:4911 , piece_id=AFGDPAP7IIQJ6PKZ34KXHG2GOUWQ7AIM3E6NQ4O56Z7QOGNUWLRQ, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 188.166.225.170:38878->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 00:29:31.908598877 (84h53m41.631161918s) [noise=true , address=104.36.82.9:4005 , piece_id=GI5GLJGPEXKPGJCU67P5QIJ5Q2OR5QVG2RGB7LQSWCWKLTOHPKRA, seg_pos={0 0}, upload=81457]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: context deadline exceeded; context deadline exceeded Jul 28 00:29:31.908613525 (84h53m41.631176565s) [seg_pos={0 0}, upload=81457]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 00:29:32.144150239 (84h53m41.866713319s) [upload=81457]: Done uploading segments... Jul 28 00:29:32.144183573 (84h53m41.866746653s) [upload=81457]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841350662s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............................X.......X.XX....X....X.......] duration: 5.717905685s FAIL (2025-07-28 00:34:32 UTC) orbiter warning: digest mismatch: after 6b93ca9819ba9c70b065b54057b3f984abfbe2ed3f41c7f9ca09ec5e2ec8ba56 != before bfb4991c3c789f3b76f90db2ddfb43319ad13805292a8a70202450851dea6594 Duration: 5.717930061s core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)