Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.012138802s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:29:48 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.012174119s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
8 out of 60 is failed (threshold=3) [..............X...X...........XX.....X...X..............X..X] duration:
1m0.00177653s trace:
https://jaeger.dev.storj.io/trace/62c9a37f0753838d
FAIL (2025-12-09 03:29:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001790176s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
273.749891ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/b66aeadd70f1f54
FAIL (2025-12-09 03:29:34 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout
Duration: 273.778235ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [........X...X.........XX.X.............X............X.......] duration:
3.441660281s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:29:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.441686682s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [........................XX..................................] duration:
237.914584ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:29:32 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 237.938058ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
4.894162115s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:29:29 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.894188725s
core
uplink
eu1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [........................XX..................................] duration:
1.160157707s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:29:23 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.160185709s
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
6.202144066s trace:
https://jaeger.dev.storj.io/trace/5f4ccc758f414b33
FAIL (2025-12-09 03:29:19 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.202174325s
core
uplink
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..........................X.................................] duration:
260.223844ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:28:50 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 260.246018ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.456584201s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:28:49 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.456762121s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [.........................XX.................................] duration:
252.752066ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:28:41 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 252.780771ms
edge
linksharing
bgp
eu1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [.........................XX.................................] duration:
1.152207179s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:28:40 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.152234429s
edge
authservice
http
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
267.350186ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/b66aeadd70f1f54
FAIL (2025-12-09 03:28:24 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout
Duration: 267.377209ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.........X...X.........XX.X.............X............X......] duration:
3.479920716s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:28:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.479945283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.752622753s trace:
https://jaeger.dev.storj.io/trace/5f4ccc758f414b33
FAIL (2025-12-09 03:28:19 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.752648783s
core
uplink
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [..........................XX................................] duration:
240.188714ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:28:14 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 240.221857ms
edge
linksharing
bgp
eu1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [..........................XX................................] duration:
1.177704388s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:28:13 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.177733073s
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
4.953205424s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:28:06 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 4.953233919s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
263.966332ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/b66aeadd70f1f54
FAIL (2025-12-09 03:28:06 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout
Duration: 263.999875ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..........X...X.........XX.X.............X............X.....] duration:
4.025547542s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:27:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.025576626s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...........................X................................] duration:
294.960008ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:27:50 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 294.986589ms
edge
linksharing
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [...........X...X.........XX.X.............X............X....] duration:
3.404847606s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:27:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.405016407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [...............X...X...........XX.....X...X..............X..] duration:
5.901541885s trace:
https://jaeger.dev.storj.io/trace/2c91eb3240b089d3
FAIL (2025-12-09 03:27:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.901562335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.068200116s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:27:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.068224742s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [............................X...............................] duration:
277.036909ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:27:21 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 277.065564ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [...........................XX...............................] duration:
235.898235ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:27:20 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 236.523757ms
edge
linksharing
bgp
eu1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [...........................XX...............................] duration:
1.156654054s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:27:19 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.156685023s
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.35660562s trace:
https://jaeger.dev.storj.io/trace/5f4ccc758f414b33
FAIL (2025-12-09 03:27:15 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.356624035s
core
uplink
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
291.728797ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/b66aeadd70f1f54
FAIL (2025-12-09 03:27:14 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout
Duration: 291.783149ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [................X...X...........XX.....X...X..............X.] duration:
5.965316252s trace:
https://jaeger.dev.storj.io/trace/2c91eb3240b089d3
FAIL (2025-12-09 03:27:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.965340347s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [............X...X.........XX.X.............X............X...] duration:
3.422092199s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:26:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.422120082s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.041575357s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:26:55 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.041607607s
core
uplink
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
7 out of 60 is failed (threshold=3) [.................X...X...........XX.....X...X..............X] duration:
1m0.001406706s trace:
https://jaeger.dev.storj.io/trace/2c91eb3240b089d3
FAIL (2025-12-09 03:26:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001427526s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [............................XX..............................] duration:
249.399082ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:26:52 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 249.423058ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.758295735s trace:
https://jaeger.dev.storj.io/trace/47af5af0eb589e37
FAIL (2025-12-09 03:26:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload
Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload
Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments...
Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false
Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment...
Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment.
Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment
Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments...
Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000
Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments...
Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.758354076s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.129796703s trace:
https://jaeger.dev.storj.io/trace/5f4ccc758f414b33
FAIL (2025-12-09 03:26:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.12982718s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
37.758066324s trace:
https://jaeger.dev.storj.io/trace/6a7a1dd9962a4cd4
FAIL (2025-12-09 03:26:44 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 37.758092814s
core
uplink
eu1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [.............................X..............................] duration:
259.995079ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:26:37 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 260.026869ms
edge
linksharing
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.784478822s trace:
https://jaeger.dev.storj.io/trace/47af5af0eb589e37
FAIL (2025-12-09 03:26:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload
Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload
Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments...
Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false
Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment...
Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment.
Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment
Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments...
Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000
Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments...
Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.784506705s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [..............................X.............................] duration:
259.862365ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:26:19 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 259.889527ms
edge
linksharing
bgp
us1
|
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
262.680863ms target:
https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 trace:
https://jaeger.dev.storj.io/trace/b66aeadd70f1f54
FAIL (2025-12-09 03:26:13 UTC)
orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": dial tcp 207.211.208.129:443: i/o timeout
Duration: 262.711401ms
edge
gateway
vhost
bgp
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [.............X...X.........XX.X.............X............X..] duration:
3.498610024s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:26:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.498644809s
core
uplink
us-select-1
us1-placement-20
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [............................XX..............................] duration:
1.977647468s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:26:07 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.977672165s
edge
authservice
http
bgp
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
5.675378265s trace:
https://jaeger.dev.storj.io/trace/5f4ccc758f414b33
FAIL (2025-12-09 03:26:02 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.675406649s
core
uplink
us1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [.............................XX.............................] duration:
236.461928ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:26:00 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 236.490162ms
edge
linksharing
bgp
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.697904879s trace:
https://jaeger.dev.storj.io/trace/47af5af0eb589e37
FAIL (2025-12-09 03:25:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 9 02:46:38.853868854 (83h17m38.724951899s) [upload=72743]: Starting upload
Dec 9 02:46:38.854074205 (83h17m38.72515724s) [upload=72743]: Done starting upload
Dec 9 02:46:38.854186769 (83h17m38.725269804s) [upload=72743]: Uploading segments...
Dec 9 02:46:38.854286329 (83h17m38.725369364s) [upload=72743]: Got next segment. Inline: false
Dec 9 02:46:38.854327998 (83h17m38.725411043s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 9 02:46:38.854334550 (83h17m38.725417585s) [upload=72743]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 9 02:46:39.098439245 (83h17m38.96952229s) [seg_pos={0 0}, upload=72743]: Begin upload segment...
Dec 9 02:46:39.098484080 (83h17m38.969567125s) [seg_pos={0 0}, upload=72743]: Done begin upload segment.
Dec 9 02:46:39.098507335 (83h17m38.96959037s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099083862 (83h17m38.970166897s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.099840784 (83h17m38.970923819s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100231177 (83h17m38.971314212s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100560174 (83h17m38.971643209s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.100874362 (83h17m38.971957397s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101176978 (83h17m38.972260023s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101476899 (83h17m38.972559934s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101782191 (83h17m38.972865236s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.101865329 (83h17m38.972948364s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102382584 (83h17m38.973465619s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.102980051 (83h17m38.974063086s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.103629859 (83h17m38.974712904s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104102448 (83h17m38.975185493s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104505055 (83h17m38.97558809s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:39.104896761 (83h17m38.975979806s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Uploading piece...
Dec 9 02:46:40.190383583 (83h17m40.061466618s) [noise=true , address=77.90.31.19:4841 , piece_id=TVZKLBWHARGXILJINV5TDGHSKDTNKZ5GPSAKY2V6JT4N6WI4OYGA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, address:77.90.31.19:4841): protocol: storage node overloaded, request limit: 2000
Dec 9 02:46:40.792059560 (83h17m40.663142605s) [upload=72743]: Next returned nil segment
Dec 9 02:46:40.792076081 (83h17m40.663159126s) [upload=72743]: Waiting for error group managing segments...
Dec 9 02:46:41.076648744 (83h17m40.947731789s) [noise=true , address=145.79.227.4:4185 , piece_id=X5MB4YETV5AYDBPUEDOHQORH3X44UJOB6PWHJBQEZYVASPLRIRDQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.081020118 (83h17m40.952103163s) [noise=true , address=145.79.227.24:4582 , piece_id=WHEQSAGINHCH33TFWYKV6T22I3KHZNC7RMCQOM6QF6WVGCMYKCEQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090419106 (83h17m40.961502141s) [noise=true , address=145.79.227.43:4159 , piece_id=Q6NKBAAEIA4AL5YPISTRMSNOAZCDHNRSLFT7MKQCDRC5GMEMRDYA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.090474090 (83h17m40.961557125s) [noise=true , address=145.79.227.28:4726 , piece_id=UZ2A5SYT2DKH73UJERMTIYSA5PILH4IKBPTF4WXENCQXEHQ5KFBA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.091834390 (83h17m40.962917435s) [noise=true , address=145.79.227.26:4655 , piece_id=MEVTYCQCMQCNI5DHWRM7GGKB2RA2PB6DFRZB6YLQJQE3HPROHTMA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093071466 (83h17m40.964154501s) [noise=true , address=145.79.227.27:4691 , piece_id=XJBBJZ2PWQT547KGNJ5K3NB5HBXCMUCKWOQQ6G3BRLUSVNHKNDTA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.093637243 (83h17m40.964720288s) [noise=true , address=145.79.227.31:4798 , piece_id=AM35FUML7ASXEWXXWCZMEPFLMLZVXTQEGKE3EPJU7XZ5X2YLHM2Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.106498503 (83h17m40.977581548s) [noise=true , address=145.79.227.16:4912 , piece_id=Q6TEEGATXLWKDJJJJSJJHQOIDUR7WUX2NB2UQ4AY6LEGZJRXJJYQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.110514860 (83h17m40.981597895s) [noise=true , address=145.79.227.42:4123 , piece_id=GSQLW2T7ZUPXYQO3S43WJMLUQ7AYGYDTSQ2WNCO7FCPZ64OPZN5Q, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.121401230 (83h17m40.992484265s) [noise=true , address=145.79.227.41:4085 , piece_id=GZIAWG2SZWKDBROYBXHMB4KXNBFBQOUHV3CHAHF6MOOCULQIZ2EQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.202085837 (83h17m41.073168872s) [noise=true , address=77.90.31.17:4772 , piece_id=MQYHZ3FVFWLLEDGJOHCIWKTLYFJGBACKBLNWX56QQQZKWQ6HP6SA, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:41.270030003 (83h17m41.141113058s) [noise=true , address=145.79.227.33:4870 , piece_id=732PTVSMVIQLHXTPJOKJJB3K6KEBPLFP6YWDYSG3D53S5XD5ZJNQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: <nil>
Dec 9 02:46:42.584185797 (83h17m42.455268832s) [noise=true , address=77.90.31.13:4626 , piece_id=URKC7SOTQMXVTTJU42DA42EGNQZX5K6MU76JNLUSVI6MWZK24YGQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: storage node overloaded, request limit: 2000
Dec 9 02:47:08.159339817 (83h18m8.030422862s) [noise=true , address=145.79.227.16:4910 , piece_id=2JZQWAR3SLAEMZ4H7JZ5J6C3U3AW234G6AOG7SVQ4FHHA73UMYAQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:145.79.227.16:4910): protocol: read tcp 188.166.225.170:46826->145.79.227.16:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159814941 (83h18m8.030897976s) [noise=true , address=145.79.227.9:4366 , piece_id=5BK5MV74XY26OH2SWY7L26LQ6LWKBKCYOSZVAYSS5U7IUS47W5DQ, seg_pos={0 0}, upload=72743]: Done uploading piece. err: ecclient: upload failed (node:1evHXReGUBYNCZaGnZ2PwmN4xZH1ABzorfhP7w61sHq4EK8Cpy, address:145.79.227.9:4366): protocol: read tcp 188.166.225.170:51146->145.79.227.9:4366: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 9 02:47:08.159959136 (83h18m8.031042181s) [seg_pos={0 0}, upload=72743]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 9 02:47:08.401226707 (83h18m8.272309753s) [upload=72743]: Done uploading segments...
Dec 9 02:47:08.401267145 (83h18m8.27235019s) [upload=72743]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.697945127s
core
uplink
us-select-1
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)antiblip:
2 out of 60 is failed (threshold=6) [..............................XX............................] duration:
240.910147ms target:
https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7ca1ea61a644f062
FAIL (2025-12-09 03:25:25 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 240.947798ms
edge
linksharing
bgp
eu1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
7 out of 60 is failed (threshold=6) [..............X...X.........XX.X.............X............X.] duration:
3.415746648s trace:
https://jaeger.dev.storj.io/trace/16ce98f337fdc6b3
FAIL (2025-12-09 03:25:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.415769972s
core
uplink
us-select-1
us1-placement-20
|
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.us1.linksharing)antiblip:
1 out of 60 is failed (threshold=6) [...............................X............................] duration:
268.066138ms target:
https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt trace:
https://jaeger.dev.storj.io/trace/7fd16448e790267f
FAIL (2025-12-09 03:25:17 UTC)
orbiter warning: Get "https://link.storjshare.io/raw/jw7f74fzbtwblz3mwqolmnivikxq/uptime-checks/hello-world.txt": dial tcp 207.211.208.129:443: i/o timeout
Duration: 268.094854ms
edge
linksharing
bgp
us1
|
checkers.AuthServiceHTTP{"https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us1)antiblip:
2 out of 60 is failed (threshold=3) [.............................XX.............................] duration:
1.175551327s trace:
https://jaeger.dev.storj.io/trace/407d3ea9756acb8a
FAIL (2025-12-09 03:25:16 UTC)
orbiter warning: Post "https://auth.storjshare.io/v1/access": dial tcp 207.211.208.129:443: i/o timeout
Duration: 1.175577288s
edge
authservice
http
bgp
us1
|