Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.78479476s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 23:01:19 UTC) orbiter warning: context deadline exceeded Duration: 5.784819316s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.077855628s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 23:00:13 UTC) orbiter warning: context deadline exceeded Duration: 6.077875466s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.92095698s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:59:27 UTC) orbiter warning: context deadline exceeded Duration: 5.920982229s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.967430025s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:59:08 UTC) orbiter warning: context deadline exceeded Duration: 5.967459891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.86410716s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:58:38 UTC) orbiter warning: context deadline exceeded Duration: 5.86413322s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.722477146s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:57:46 UTC) orbiter warning: context deadline exceeded Duration: 5.722510338s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.056261228s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:57:14 UTC) orbiter warning: context deadline exceeded Duration: 6.056291596s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.934039175s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:56:57 UTC) orbiter warning: context deadline exceeded Duration: 5.9340678s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 6.010869113s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:56:22 UTC) orbiter warning: context deadline exceeded Duration: 6.010885184s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.875446611s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:56:06 UTC) orbiter warning: context deadline exceeded Duration: 5.875482891s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.356714959s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:55:37 UTC) orbiter warning: context deadline exceeded Duration: 7.356742892s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.32929611s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:54:50 UTC) orbiter warning: context deadline exceeded Duration: 6.329324794s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.769354375s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:53:53 UTC) orbiter warning: context deadline exceeded Duration: 5.769373762s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.080655429s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:52:59 UTC) orbiter warning: context deadline exceeded Duration: 6.080690415s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.881890882s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:52:11 UTC) orbiter warning: context deadline exceeded Duration: 5.88192205s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.700949275s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:51:49 UTC) orbiter warning: context deadline exceeded Duration: 5.700987508s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.600392473s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:51:13 UTC) orbiter warning: context deadline exceeded Duration: 5.600471303s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.844473431s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:50:55 UTC) orbiter warning: context deadline exceeded Duration: 5.844496837s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.195221366s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:50:13 UTC) orbiter warning: context deadline exceeded Duration: 6.195294966s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.146686331s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:49:08 UTC) orbiter warning: context deadline exceeded Duration: 6.146740033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.757155511s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:48:20 UTC) orbiter warning: context deadline exceeded Duration: 5.757170549s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.088567363s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:47:24 UTC) orbiter warning: context deadline exceeded Duration: 6.088594665s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.913187837s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:46:32 UTC) orbiter warning: context deadline exceeded Duration: 5.91325304s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 6.193287294s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:45:35 UTC) orbiter warning: context deadline exceeded Duration: 6.193309246s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 7.768454652s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:45:16 UTC) orbiter warning: context deadline exceeded Duration: 7.768498947s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.96679704s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:44:29 UTC) orbiter warning: context deadline exceeded Duration: 5.966826236s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.027227052s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:43:23 UTC) orbiter warning: context deadline exceeded Duration: 6.027265334s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.214567412s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:42:29 UTC) orbiter warning: context deadline exceeded Duration: 6.214601807s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001067736s trace: https://jaeger.dev.storj.io/trace/6a17ae35ec025428 FAIL (2025-11-25 22:42:10 UTC) orbiter warning: context deadline exceeded Duration: 30.00109082s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.135324778s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:08:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.13535651s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.94012627s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:07:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.940154325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.139445425s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:06:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.139471947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.035275453s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:06:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.035303967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.82362357s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:04:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.823655612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.793804761s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:04:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.793847292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.901098758s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:03:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.901129095s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.903962677s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:02:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.904004176s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.841235722s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:01:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.841261653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.827036786s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:00:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.827072623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.981788758s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 21:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.981824276s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.799581684s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:59:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.799617653s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.646175904s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:57:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.646217461s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.897145087s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:56:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.897173962s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.709975137s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:56:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.710019421s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.661570116s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:56:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.661607338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.885142279s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:55:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.885170764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.926643087s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:54:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.926667292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.967021627s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:53:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.967050923s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.705039827s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:52:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.705070404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.704655729s trace: https://jaeger.dev.storj.io/trace/5642ed7acd122950 FAIL (2025-11-25 20:51:19 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 25 20:20:38.008249883 (396h16m36.559720344s) [upload=348979]: Starting upload Nov 25 20:20:38.008329035 (396h16m36.559799496s) [upload=348979]: Done starting upload Nov 25 20:20:38.008498057 (396h16m36.559968628s) [upload=348979]: Uploading segments... Nov 25 20:20:38.008578069 (396h16m36.56004853s) [upload=348979]: Got next segment. Inline: false Nov 25 20:20:38.008614799 (396h16m36.56008526s) [upload=348979]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 20:20:38.008621020 (396h16m36.560091481s) [upload=348979]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 20:20:38.255571717 (396h16m36.807042168s) [seg_pos={0 0}, upload=348979]: Begin upload segment... Nov 25 20:20:38.255638273 (396h16m36.807108734s) [seg_pos={0 0}, upload=348979]: Done begin upload segment. Nov 25 20:20:38.255689741 (396h16m36.807160202s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.255718596 (396h16m36.807189058s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256364436 (396h16m36.807834897s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.256703101 (396h16m36.808173552s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257065782 (396h16m36.808536243s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257390079 (396h16m36.80886054s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.257945657 (396h16m36.809416128s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258300964 (396h16m36.809771425s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.258685586 (396h16m36.810156037s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.259076129 (396h16m36.81054658s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.261934522 (396h16m36.813404983s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.266452646 (396h16m36.817923107s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.269742521 (396h16m36.821212982s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.273490707 (396h16m36.824961168s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.280541323 (396h16m36.832011784s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:38.281122690 (396h16m36.832593142s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Uploading piece... Nov 25 20:20:39.361391324 (396h16m37.912861795s) [noise=true , address=77.90.31.19:4841 , piece_id=IFHRGZIG2XNXMN3XB6EZV3OMVXXNUDF2QAKAPTXYVLF4WXVQR7MQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000 Nov 25 20:20:39.868243374 (396h16m38.419713835s) [upload=348979]: Next returned nil segment Nov 25 20:20:39.868263021 (396h16m38.419733482s) [upload=348979]: Waiting for error group managing segments... Nov 25 20:20:40.251250099 (396h16m38.80272056s) [noise=true , address=145.79.227.42:4123 , piece_id=GH56P7YLKHLQFI4345V23XWXDJJ6TRRNG4JCC4F4MXAOJFRAS27A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.253693672 (396h16m38.805164134s) [noise=true , address=145.79.227.11:4730 , piece_id=6G6W752HUJY7MPVWDSLGLULWSNC2V6CUM5P6N76V774UCKDXIJOA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.278361104 (396h16m38.829831555s) [noise=true , address=145.79.227.13:4803 , piece_id=QQTXKEUMYK6I2PISSWMTZXWBUJT2GZ7LUVQZ4CSXCL6P2NENXO7Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.287226495 (396h16m38.838696958s) [noise=true , address=145.79.227.17:4969 , piece_id=TH7PPK2J4VYPEYY3P4VJCX2P2OYAXRFNOOR3EDQGH7QWXP6244HQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.303893843 (396h16m38.855364294s) [noise=true , address=145.79.227.29:4941 , piece_id=FB3N43GLPA5AGFPTTD43TIWOF33J6CMA2GGIS3B62MQNGH2W7NDA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.460969677 (396h16m39.012440138s) [noise=true , address=45.91.116.7:4009 , piece_id=VHQKBKP6FAAV6B36V3KN6WJD3TH4K6C6CIOESBM6FPHI7IB4W5GQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:40.553165077 (396h16m39.104635538s) [noise=true , address=77.90.31.18:4807 , piece_id=HMOV5WHJ3S3Y4VPD674RZ5AWDLCLD4AJTTO2BZDKR3DW6M6GPD6Q, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:43.098251148 (396h16m41.649721599s) [noise=true , address=145.79.227.24:4582 , piece_id=LF2OQV6AE4GMY4GEYVJGLDLJ5RXCNUSLMI5JIFOMMGGOBYWNUQ2A, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:50.140334338 (396h16m48.691804799s) [noise=true , address=145.79.227.33:4871 , piece_id=LVQWQXIRSZC72XCN5YLZZCF7NFNDGBZI7SGZGXESW35X4TCMMNOQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:20:54.579450008 (396h16m53.130920469s) [noise=true , address=145.79.227.32:4834 , piece_id=P2FXRA44GGHN4AP4CL7IXAZV4FJPIYUPGMQTTDLU5O7B2NE67JDQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:01.780024853 (396h17m0.331495314s) [noise=true , address=145.79.227.16:4909 , piece_id=RD44KK7RJSPJBEVM6QVA3OHOJ7VK74LG7RGNFVXO7JBQHBZUX6VQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:03.768691675 (396h17m2.320162136s) [noise=true , address=145.79.227.14:4838 , piece_id=OYNZQ2IFDI4EVHWDZ5N73AR2P7G25BMK32F4CIDR2MEA2U6WXCUQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: <nil> Nov 25 20:21:07.314778253 (396h17m5.866248724s) [noise=true , address=145.79.227.38:4978 , piece_id=6TSXWEZDXYCRDEPJIK3GFKTFPEKHQUAK2CXE7ZYLXDU7SBRHJRMA, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45800->145.79.227.38:4978: use of closed network connection Nov 25 20:21:07.314910134 (396h17m5.866380595s) [noise=true , address=145.79.227.9:4366 , piece_id=FTQDNOKON7MVFPNT4BIF4UKXHWQPDXKN7ZTLVOR423VGXOTJXYHQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42334->145.79.227.9:4366: use of closed network connection Nov 25 20:21:07.315269659 (396h17m5.86674012s) [noise=true , address=145.79.227.2:4115 , piece_id=LF2KQTSNJL7YRIBKGCME3G27ENDE47QIEN3CVB5TCRCJV7WLSVIQ, seg_pos={0 0}, upload=348979]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 20:21:07.315305938 (396h17m5.866776409s) [seg_pos={0 0}, upload=348979]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 25 20:21:07.554555289 (396h17m6.10602575s) [upload=348979]: Done uploading segments... Nov 25 20:21:07.554603681 (396h17m6.106074142s) [upload=348979]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.704681717s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)