Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
1.376566116s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:35:14 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.376599971s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
1.416325715s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:34:16 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.416358777s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
1.436898891s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:33:45 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.436929861s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
1.382327315s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:32:47 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.382353705s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
1.369459431s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:32:03 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 1.369491843s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
3.151841504s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:31:08 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 3.151883654s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.eu1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001828211s trace:
https://jaeger.dev.storj.io/trace/292b9465c989993a
FAIL (2025-12-23 08:30:49 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 30.001855744s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
248.946473ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:21:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 248.972935ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
262.316952ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:21:01 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 262.337183ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
243.542924ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:19:53 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 243.5617ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
253.845853ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:19:24 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 253.866772ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
255.838207ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:19:03 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 255.866761ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
250.403367ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:18:42 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 250.429979ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
254.196259ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:17:52 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 254.220235ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
264.965528ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:16:44 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 264.991587ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
244.549962ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:16:02 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 244.694829ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
245.574372ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:15:12 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 245.607536ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
252.120217ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:14:20 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 252.140275ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
245.013815ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:13:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 245.040705ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
247.655624ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:13:08 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 247.678817ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
258.89033ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:12:48 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 258.917591ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
245.94093ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:12:04 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 245.958644ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...............X............................................] duration:
243.391906ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:11:08 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 243.418165ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [................X...........................................] duration:
248.658976ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:10:43 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 248.688112ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.................X..........................................] duration:
244.840804ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:09:36 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 244.871252ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..................X.........................................] duration:
252.827702ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:08:47 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 252.856188ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...................X........................................] duration:
244.516537ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:07:59 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 244.544589ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [....................X.......................................] duration:
706.043583ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:07:02 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 706.081134ms
edge
linksharing
bgp
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.....................X......................................] duration:
247.316199ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:06:11 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 247.340615ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
27.370813797s trace:
https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7
FAIL (2025-12-23 03:05:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment
Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments...
Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces.
Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled
Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments...
Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil>
Duration: 27.370890874s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [......................X.....................................] duration:
244.897354ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:05:09 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 244.950586ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.061461445s trace:
https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9
FAIL (2025-12-23 03:04:30 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 5.06149032s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.......................X....................................] duration:
246.38711ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:04:12 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 246.411565ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.267283194s trace:
https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9
FAIL (2025-12-23 03:03:59 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 5.267311648s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.974355521s trace:
https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7
FAIL (2025-12-23 03:03:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment
Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments...
Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces.
Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled
Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments...
Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil>
Duration: 6.974386008s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [........................X...................................] duration:
241.535882ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:03:50 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 241.553825ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.955865775s trace:
https://jaeger.dev.storj.io/trace/70da446b2b2ff016
FAIL (2025-12-23 03:03:41 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46)
Duration: 5.955890911s
core
uplink
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.........................X..................................] duration:
250.601515ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:03:35 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 250.624209ms
edge
linksharing
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [X...........................................................] duration:
3.97888996s trace:
https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793
FAIL (2025-12-23 03:03:27 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.978915528s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
4.835874552s trace:
https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9
FAIL (2025-12-23 03:03:17 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 4.83590533s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.883487748s trace:
https://jaeger.dev.storj.io/trace/2c98a82e82cbeda7
FAIL (2025-12-23 03:03:09 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
KL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.700896105 (418h48m56.57197914s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701329429 (418h48m56.572412474s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.701714032 (418h48m56.572797077s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702231197 (418h48m56.573314232s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.702751507 (418h48m56.573834552s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703271658 (418h48m56.574354693s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703948437 (418h48m56.575031482s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.703969256 (418h48m56.575052291s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704457947 (418h48m56.575540982s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.704834724 (418h48m56.575917769s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.706591609 (418h48m56.577674654s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707118754 (418h48m56.578201789s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707431179 (418h48m56.578514224s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.707910952 (418h48m56.578993997s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708492299 (418h48m56.579575344s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:17:56.708774406 (418h48m56.579857441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Uploading piece...
Dec 23 02:18:00.137346621 (418h49m0.008429666s) [upload=367702]: Next returned nil segment
Dec 23 02:18:00.137360507 (418h49m0.008443552s) [upload=367702]: Waiting for error group managing segments...
Dec 23 02:18:01.750516581 (418h49m1.621599616s) [noise=true , address=64.49.13.137:4500 , piece_id=RFGLMWE4OHC7RL2NBPX6VRKG4YGRVCIMBHQFW2W5XAEE376UTVVA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.125124405 (418h49m1.99620745s) [noise=true , address=77.90.31.18:4815 , piece_id=XQDHXOY2ZQQHZZHMGLO5GKZOUJZFKKWGLDJRHJLWCFEX6PKNNHWA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:02.226869841 (418h49m2.097952876s) [noise=true , address=45.91.116.8:4060 , piece_id=TRNAVEFSFXMIN3DFMQ2DUAKKDTSZH36TCSYZXFKALUM7TCKM3Y6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:05.457222504 (418h49m5.328305539s) [noise=true , address=77.90.31.13:4651 , piece_id=H6MCF4TXI5WFQ2BBCPXL3OX6OA4NCRBIH6IBLMPYPIJBETEN6UGA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:06.079641082 (418h49m5.950724128s) [noise=true , address=45.91.116.12:4257 , piece_id=SPTNFMB3LXNY74LCASNSAZIPCOAQHO5V6SOFP27Y4IY25CHWBO2A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:09.265548070 (418h49m9.136631105s) [noise=true , address=64.49.13.139:4478 , piece_id=QUBJUJHQ6AYU6L67666PA7BHCRL2IIFOJDISOISTEK4JWABGMXIQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:10.880784395 (418h49m10.751867441s) [noise=true , address=45.91.116.14:4213 , piece_id=VLD3RNN2RLOQI2AKMHNY3G57QJPDNJ3BA5JDRRJHCHNHS6YE52JA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.686956874 (418h49m12.558039929s) [noise=true , address=145.79.227.28:4742 , piece_id=3IMQS3J42XXGHCZFMBYPDCPCJYRY3I3EUIQ2Z2JOK4TPXQQH75OQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:12.832920510 (418h49m12.704003545s) [noise=true , address=45.91.116.5:4330 , piece_id=EMSR7LFH5Q6R7YAJNH45TDQS2E4KX25Q335IFFUORN5AJJWS7W5Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:22.466644063 (418h49m22.337727098s) [noise=true , address=77.90.31.16:4740 , piece_id=KYCKJ7F7CNIXIVAZZ7CQPWAYNCMIM5RKO4IKXETP5EZNOAGKGONA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:34.649740548 (418h49m34.520823583s) [noise=true , address=64.49.13.169:4715 , piece_id=6RCBEKXRJ56IGXWT4PVSTDYRIH3UCAD22C4Z5BU33U5YIPWBEPDQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:38.250548484 (418h49m38.121631529s) [noise=true , address=145.79.227.17:4984 , piece_id=DAMTMQIEBM3I2QZTWVEGISFJ5T7I3LMZ3CTXUFXO2CS775GY4F7Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.329219400 (418h49m43.200302445s) [noise=true , address=145.79.227.23:4549 , piece_id=XH7BPLOCXKVTJQDYKIRKFXVLHIXXNJFAXCFZPSSASWLOG3GM5PEQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400788824 (418h49m43.271871859s) [noise=true , address=145.79.227.32:4848 , piece_id=NMEVAZ6HHKE2Z4Q5G4VLNPOIURXE5TXXUMLFJLURZUI4Z62W7VMA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: <nil>
Dec 23 02:18:43.400819082 (418h49m43.271902117s) [seg_pos={0 0}, upload=367702]: Segment reached optimal threshold of 14 pieces.
Dec 23 02:18:43.401090247 (418h49m43.272173302s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rWJM3vC8XrUAfVaoqVZm1qzqcG8AqxLzncExPHEak5oo3QuPA): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401100788 (418h49m43.272183823s) [noise=true , address=5.231.73.4:4030 , piece_id=M6FJSRGUNCEIBMNKLBK7HHFL4DA3PDD2MZ2GN2MJKAFSH3XRI7XQ, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401281792 (418h49m43.272364837s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12R5gJFhsmocYPNZf4yNnYqJwcwAKt9yiHmmqGMk3btzVjkFV4S): protocol: expected piece hash; context canceled; EOF
Dec 23 02:18:43.401290889 (418h49m43.272373925s) [noise=true , address=77.90.31.14:4667 , piece_id=5ZXKL7BLGSD3TCZNAHNK7STV2RKAZQWCHN6PROBXAPTA5ZBCPT6A, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401412151 (418h49m43.272495186s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TxjP4CnxyU4t2DTNBjBMBxnizcHCD42fMrB5Da6Qiqg1Eg17V): protocol: context canceled; context canceled
Dec 23 02:18:43.401418663 (418h49m43.272501698s) [noise=true , address=145.79.227.13:4811 , piece_id=KVHW3GEE4KWLOOUJBRVOOLDA3EGPRUU7VHPH7TQXLURUCCU5J77Q, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401630526 (418h49m43.272713561s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pKJiP9crUtUHuDmbJhbfbuvZDDpfsXJ17N7LJUE3KDMUGMCkk): protocol: read tcp 188.166.225.170:44484->145.79.227.30:4796: use of closed network connection; conn closed: op; piecestore: piecestore close: context canceled
Dec 23 02:18:43.401638241 (418h49m43.272721276s) [noise=true , address=145.79.227.30:4796 , piece_id=IF6NPIC5DTTMJ4DZMIDD6SFN7TGDBN6RD3H7KT7A4OC7VJVCEZOA, seg_pos={0 0}, upload=367702]: Long tail cancellation due to optimal threshold
Dec 23 02:18:43.401657227 (418h49m43.272740273s) [seg_pos={0 0}, upload=367702]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 23 02:18:43.401864953 (418h49m43.272947988s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 23 02:18:43.401870614 (418h49m43.272953649s) [upload=367702]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 23 02:18:43.703068278 (418h49m43.574151323s) [upload=367702]: Done uploading segments...
Dec 23 02:18:43.703144954 (418h49m43.574227999s) [upload=367702]: Upload finished. err: <nil>
Duration: 6.88354102s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
3.56314986s trace:
https://jaeger.dev.storj.io/trace/20c185d15563c01c
FAIL (2025-12-23 03:03:07 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 3.563182722s
core
uplink
ap1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.X..........................................................] duration:
3.602288203s trace:
https://jaeger.dev.storj.io/trace/7f7fb60dfbab7793
FAIL (2025-12-23 03:03:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.602317677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.942892848s trace:
https://jaeger.dev.storj.io/trace/70da446b2b2ff016
FAIL (2025-12-23 03:03:01 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 46)
Duration: 5.942923316s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.730383063s trace:
https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24
FAIL (2025-12-23 03:03:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 2)
======LOGS======
7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment
Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments...
Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil>
Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil>
Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil>
Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments...
Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2)
Duration: 5.730408931s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.185583581s trace:
https://jaeger.dev.storj.io/trace/61a0842c4fdd78ca
FAIL (2025-12-23 03:02:53 UTC)
orbiter warning: context deadline exceeded
Duration: 6.185614169s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
3.202961808s trace:
https://jaeger.dev.storj.io/trace/20c185d15563c01c
FAIL (2025-12-23 03:02:46 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 39)
Duration: 3.202991714s
core
uplink
ap1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
246.161981ms remote:
"tcp:207.211.208.129:443" target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/1e922f3feb1984b6
FAIL (2025-12-23 03:02:28 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded
Duration: 246.187499ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
4.992927792s trace:
https://jaeger.dev.storj.io/trace/14f27c56f52ae4e9
FAIL (2025-12-23 03:02:26 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 4.992971305s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
9.922835166s trace:
https://jaeger.dev.storj.io/trace/5b87e1ad666dcc24
FAIL (2025-12-23 03:02:24 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 2)
======LOGS======
7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.480444685 (418h49m5.35152773s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.480943264 (418h49m5.352026309s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.482473688 (418h49m5.353556733s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.482598135 (418h49m5.35368118s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.485571146 (418h49m5.356654182s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.488437175 (418h49m5.35952021s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.491243618 (418h49m5.362326653s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.492801755 (418h49m5.36388479s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.495283631 (418h49m5.366366666s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.495702669 (418h49m5.366785704s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:05.499769763 (418h49m5.370852808s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Uploading piece...
Dec 23 02:18:08.589264332 (418h49m8.460347367s) [upload=367705]: Next returned nil segment
Dec 23 02:18:08.589286223 (418h49m8.460369258s) [upload=367705]: Waiting for error group managing segments...
Dec 23 02:18:12.569357328 (418h49m12.440440364s) [noise=true , address=45.91.116.4:4937 , piece_id=IIV6VFBKACIVI3M6A6L2WSDU54D3MRGZ36LG6E3E4SXKQVRX74LQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil>
Dec 23 02:18:15.106396805 (418h49m14.977479851s) [noise=true , address=77.90.31.12:4589 , piece_id=RD5KILZQDADOR2UPRRAO3XM7YWI7AQ57HBR37VC44DNZPOS555SA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000
Dec 23 02:18:17.598569766 (418h49m17.469652801s) [noise=true , address=77.90.31.16:4735 , piece_id=XEZKWPQDBCT7YRGYTH4TFR4BDCP34BPJB6OVFTYGMQ2NQI34NVZA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: <nil>
Dec 23 02:18:33.434728164 (418h49m33.305811199s) [noise=true , address=145.79.227.42:4121 , piece_id=ICOCUSNLYIVKOJTONNBAEPWGSRTB44H2KAHESVBGGBEDOEHROYGA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh, address:145.79.227.42:4121): protocol: read tcp 188.166.225.170:50058->145.79.227.42:4121: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.434850206 (418h49m33.305933241s) [noise=true , address=145.79.227.15:4874 , piece_id=PWRV5UFUDENFBPJFK3KYNXMYLQZBHFG6KBJFZXA6P32NT7ZNOWDA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:145.79.227.15:4874): protocol: read tcp 188.166.225.170:50240->145.79.227.15:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.434955458 (418h49m33.306038493s) [noise=true , address=145.79.227.50:4373 , piece_id=PZRIRRQQETAX56EVN7LY3Z5DUPUYUE7N7H2ZTYWK7Y7UWG6NFFSQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12AQGSkJuNJDxoeqhChUenWxHwF5yK41UESCF63FQz3XvabXXfz, address:145.79.227.50:4373): protocol: read tcp 188.166.225.170:54252->145.79.227.50:4373: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435015211 (418h49m33.306098246s) [noise=true , address=145.79.227.9:4366 , piece_id=73FK36X6U4CYXVKSGCTLK4YCL2QB3CQKUST2IY4OQGG4JWMLTV6A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: rpc: dial tcp 145.79.227.9:4366: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435058384 (418h49m33.306141429s) [noise=true , address=145.79.227.43:4159 , piece_id=WUJFSGFHRK4PHRLS4H3MNT5TVXYR6MSTUQXRN6QBXI47WRPKKN4A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA, address:145.79.227.43:4159): protocol: read tcp 188.166.225.170:49750->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435156329 (418h49m33.306239374s) [noise=true , address=145.79.227.47:4267 , piece_id=CFW6HHUK6IP232DVR64FPSBPULEIVPVGCS2HR2523RAIP5N77POQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1ektWku5yW6MRm2Rph8kiAAuifJJWZUMopaA3gcwStUfY6Nax8, address:145.79.227.47:4267): protocol: read tcp 188.166.225.170:33996->145.79.227.47:4267: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435495826 (418h49m33.306578871s) [noise=true , address=77.90.31.21:4877 , piece_id=J3XPMHNIGDTRPYW2ITOOW4FRJ4SUKMCCGRP22WKD55A7OA7EOV3A, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:35802->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435680898 (418h49m33.306763943s) [noise=true , address=145.79.227.41:4086 , piece_id=KXPKPLKHLFQ5GU7Z7LM3NXGWGXSGISKBOHAWPJ7QGG4ESA64YYWA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm, address:145.79.227.41:4086): protocol: read tcp 188.166.225.170:33906->145.79.227.41:4086: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435701508 (418h49m33.306784543s) [noise=true , address=145.79.227.39:4015 , piece_id=UAGG4FHO2SDQKDF6T4QIGVN7IMNANYWTD3SHYRMBVGCPS3Z4VIYA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:18etsqD4wpYunpwdbPETCAn3JGJawbFAK9mwTkVUYKuWB7ypQE, address:145.79.227.39:4015): protocol: read tcp 188.166.225.170:45508->145.79.227.39:4015: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435883414 (418h49m33.306966449s) [noise=true , address=145.79.227.37:4565 , piece_id=PAKR2BG4S3KYFXOLS7BL5AANJMIHMN477KGMJJ4XIHMTF7UOTCHA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1nvvNczHtQrSEJuJbErnjiAWa4BRDzBrQmBNbB7RrrqN1JWNHT, address:145.79.227.37:4565): protocol: read tcp 188.166.225.170:43228->145.79.227.37:4565: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.435913952 (418h49m33.306996987s) [noise=true , address=145.79.227.38:4978 , piece_id=5VMIQGZ32DI7IE6Z6HMVCJUXIMBFVN6BWONEBE452PG5WDWZXDZQ, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12PcGHVSm5UAXPh5vCfyRHCd7Bdb67oXF51AZcnRFUjXqVYzXLR, address:145.79.227.38:4978): protocol: read tcp 188.166.225.170:57482->145.79.227.38:4978: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436073857 (418h49m33.307156892s) [noise=true , address=145.79.227.35:4481 , piece_id=EVLROBIPMCRCZYYUSUKKCHCMTJH6G2H56QJR6ZPIF7R7HVVCP6CA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:145.79.227.35:4481): protocol: read tcp 188.166.225.170:34156->145.79.227.35:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436167987 (418h49m33.307251022s) [noise=true , address=145.79.227.51:4411 , piece_id=2JQUVX7JG3GYCRYJMYE7AQNIX7XG7XEYRXAC6JTTYRM343CVP3IA, seg_pos={0 0}, upload=367705]: Done uploading piece. err: ecclient: upload failed (node:1DT68m6J4PpDk3XUDhDaaqV46NLKVz4A23716JLFxEAh122GeR, address:145.79.227.51:4411): protocol: read tcp 188.166.225.170:54092->145.79.227.51:4411: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 23 02:18:33.436191531 (418h49m33.307274566s) [seg_pos={0 0}, upload=367705]: Done waiting for segment. successful: 2 optimal: 13 errs: <nil>
Dec 23 02:18:33.726571837 (418h49m33.597654882s) [upload=367705]: Done uploading segments...
Dec 23 02:18:33.726612455 (418h49m33.59769549s) [upload=367705]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 2)
Duration: 9.922877418s
core
uplink
us-select-1
|