Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [X.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXXX] duration:
5.001480005s trace:
https://jaeger.dev.storj.io/trace/20bc1c6c60b8521
FAIL (2025-12-05 13:50:12 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001512686s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X................................X..X.......] duration:
26.458793941s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:50:01 UTC)
orbiter warning: context deadline exceeded
Duration: 26.458822886s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [XX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXXX] duration:
5.001197665s trace:
https://jaeger.dev.storj.io/trace/791960763d7e955
FAIL (2025-12-05 13:49:50 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0012208s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
12.726332831s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:49:23 UTC)
orbiter warning: context deadline exceeded
Duration: 12.72636343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
47 out of 60 is failed (threshold=6) [XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXXX] duration:
5.000635085s trace:
https://jaeger.dev.storj.io/trace/70514dc455eef0b6
FAIL (2025-12-05 13:48:47 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000689569s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X................................X..X......] duration:
16.045657828s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:48:41 UTC)
orbiter warning: context deadline exceeded
Duration: 16.045687064s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
13.134399301s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:48:38 UTC)
orbiter warning: context deadline exceeded
Duration: 13.134427775s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X................................X..X.....] duration:
22.828160593s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:48:22 UTC)
orbiter warning: context deadline exceeded
Duration: 22.828190732s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...................X................................X..X....] duration:
20.051282639s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:47:35 UTC)
orbiter warning: context deadline exceeded
Duration: 20.051305904s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXXX] duration:
5.001725855s trace:
https://jaeger.dev.storj.io/trace/ab183375a58c7c1
FAIL (2025-12-05 13:47:34 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001751195s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
12.045493033s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:47:27 UTC)
orbiter warning: context deadline exceeded
Duration: 12.045522929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
46 out of 60 is failed (threshold=6) [X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXXX] duration:
5.000754386s trace:
https://jaeger.dev.storj.io/trace/1c788409762a0698
FAIL (2025-12-05 13:47:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000845159s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
24.166930116s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:47:07 UTC)
orbiter warning: context deadline exceeded
Duration: 24.166961987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................X................................X..X...] duration:
16.325793445s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:46:29 UTC)
orbiter warning: context deadline exceeded
Duration: 16.325819474s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
45 out of 60 is failed (threshold=6) [.X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXXX] duration:
5.001058887s trace:
https://jaeger.dev.storj.io/trace/6fc97ae3b58c41bf
FAIL (2025-12-05 13:46:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001088904s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
15.645416249s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:45:44 UTC)
orbiter warning: context deadline exceeded
Duration: 15.645441217s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXXX] duration:
5.001911869s trace:
https://jaeger.dev.storj.io/trace/7de68ce54922e0b2
FAIL (2025-12-05 13:45:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001934411s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................X................................X..X..] duration:
20.732575361s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:45:19 UTC)
orbiter warning: context deadline exceeded
Duration: 20.732604126s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
16.213498268s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:45:13 UTC)
orbiter warning: context deadline exceeded
Duration: 16.213538484s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXXX] duration:
5.001067813s trace:
https://jaeger.dev.storj.io/trace/6dff97232708d621
FAIL (2025-12-05 13:45:06 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001094855s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXXX] duration:
5.001159838s trace:
https://jaeger.dev.storj.io/trace/4e2cc26f03d0d75a
FAIL (2025-12-05 13:44:25 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001182342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......................X................................X..X.] duration:
19.009256708s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:44:15 UTC)
orbiter warning: context deadline exceeded
Duration: 19.009282078s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXXX] duration:
5.000785095s trace:
https://jaeger.dev.storj.io/trace/16736007fece8869
FAIL (2025-12-05 13:44:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000894753s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
16.721566577s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:44:07 UTC)
orbiter warning: context deadline exceeded
Duration: 16.721633874s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................X................................X..X] duration:
30.001504244s trace:
https://jaeger.dev.storj.io/trace/27a08c49f32ebf99
FAIL (2025-12-05 13:43:53 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001530645s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001307899s trace:
https://jaeger.dev.storj.io/trace/38cafe1bf9e21fa6
FAIL (2025-12-05 13:43:50 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001327828s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.679596119s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:42:57 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.679626948s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXXX] duration:
5.001618132s trace:
https://jaeger.dev.storj.io/trace/56cd46dfbb4d20c2
FAIL (2025-12-05 13:42:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001665732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................X................................X..] duration:
17.771610257s trace:
https://jaeger.dev.storj.io/trace/3c8d96cbdc74aa48
FAIL (2025-12-05 13:42:55 UTC)
orbiter warning: context deadline exceeded
Duration: 17.771641076s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.298770635s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:42:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.29880488s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXXX] duration:
5.001590238s trace:
https://jaeger.dev.storj.io/trace/1f185de8983d515b
FAIL (2025-12-05 13:42:14 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001628601s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXXX] duration:
5.002291033s trace:
https://jaeger.dev.storj.io/trace/b82c8060d6d724e
FAIL (2025-12-05 13:41:58 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002313475s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X................................X.] duration:
23.903754078s trace:
https://jaeger.dev.storj.io/trace/3c8d96cbdc74aa48
FAIL (2025-12-05 13:41:55 UTC)
orbiter warning: context deadline exceeded
Duration: 23.90378204s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.160087504s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:41:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 6.160118884s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [.XXX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXXX] duration:
5.002386485s trace:
https://jaeger.dev.storj.io/trace/5504ce4f23424df8
FAIL (2025-12-05 13:41:28 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002412925s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
7.231247237s trace:
https://jaeger.dev.storj.io/trace/275a4a7f39135e2a
FAIL (2025-12-05 13:40:59 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 7.231275902s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.83159377s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:40:51 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.831624911s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X................................X] duration:
30.001313129s trace:
https://jaeger.dev.storj.io/trace/3c8d96cbdc74aa48
FAIL (2025-12-05 13:40:49 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001331335s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
41 out of 60 is failed (threshold=6) [..XXX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXXX] duration:
5.000862865s trace:
https://jaeger.dev.storj.io/trace/13848eb6be74ca6a
FAIL (2025-12-05 13:40:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000886279s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.601045693s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:40:00 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.601085088s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
7.198109684s trace:
https://jaeger.dev.storj.io/trace/275a4a7f39135e2a
FAIL (2025-12-05 13:39:45 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 7.198150513s
core
uplink
us1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
15.3990771s trace:
https://jaeger.dev.storj.io/trace/70ddcc9c759b9c09
FAIL (2025-12-05 13:39:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.399099713s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [...XXX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XXX] duration:
5.001578454s trace:
https://jaeger.dev.storj.io/trace/63c64ad097304188
FAIL (2025-12-05 13:39:43 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001599856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.192567305s trace:
https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb
FAIL (2025-12-05 13:39:34 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.192608524s
core
uplink
eu1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
22.214394105s trace:
https://jaeger.dev.storj.io/trace/70ddcc9c759b9c09
FAIL (2025-12-05 13:39:28 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 22.214444221s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
5.931588599s trace:
https://jaeger.dev.storj.io/trace/275a4a7f39135e2a
FAIL (2025-12-05 13:39:21 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 5.931609719s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.786487369s trace:
https://jaeger.dev.storj.io/trace/16d0fd0d28b4b6f7
FAIL (2025-12-05 13:39:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 24)
======LOGS======
seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.667348134 (44h2m9.138558863s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.668113411 (44h2m9.13932414s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.669515572 (44h2m9.140726301s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670009011 (44h2m9.14121974s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670629353 (44h2m9.141840092s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.670996271 (44h2m9.142207s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.671737944 (44h2m9.142948673s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674169844 (44h2m9.145380573s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.674772422 (44h2m9.145983151s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675359480 (44h2m9.146570219s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:32.675987596 (44h2m9.147198335s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Uploading piece...
Dec 5 12:57:33.888855747 (44h2m10.360066487s) [upload=38726]: Next returned nil segment
Dec 5 12:57:33.888874513 (44h2m10.360085252s) [upload=38726]: Waiting for error group managing segments...
Dec 5 12:57:34.458992275 (44h2m10.930203034s) [noise=true , address=81.171.14.70:4036 , piece_id=G67IW7YF7VX5GRSOUEJECEN66A5HMVZAGCNOEMJ6J76Q3WQCJDBQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.463924437 (44h2m10.935135166s) [noise=true , address=81.171.14.70:4042 , piece_id=V2B6EAOBLNVENCOKBKQHCU5ZAE2L33LSAUSMFCWPXBRSFGS5W7PQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.466804100 (44h2m10.938014849s) [noise=true , address=81.171.14.70:4028 , piece_id=ZQXS4NQTFFY2NJLE6E2CMHFOEVXKXTE5B7WC6EHQNOLZQLBF43KA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.482112661 (44h2m10.9533234s) [noise=true , address=81.171.14.70:4047 , piece_id=4G7U2PYPMWT7QHMFMKWDNIXKDQWPRQBR32R74MD6IOXTFOCGAMJQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.503767315 (44h2m10.974978044s) [noise=true , address=81.171.14.70:4026 , piece_id=6UGKVAGWTYIJ6PKDQOVGBN6TJMKKOT3XAJDJXL6VQ64RLX434Q2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.534018609 (44h2m11.005229339s) [noise=true , address=81.171.14.70:4039 , piece_id=GB3FEVE6G4SHK4VVFKEIH5FV43PGPGCV3BXUSZQQ6CGVTY6EY2MA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.926473649 (44h2m11.397684378s) [noise=true , address=81.171.14.70:4037 , piece_id=6ZNMEGPIZOYN4KQ4N3HKRA3ZTDOE3RQA4MK2BWS52LTN6G2UVTQA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:34.996303367 (44h2m11.467514097s) [noise=true , address=81.171.14.70:4044 , piece_id=PLPPT6RYM734KYQ2NHOJ3PQRXCRZPVGLSFNKGTEDPDV2HUT4IV7Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.099023697 (44h2m11.570234426s) [noise=true , address=89.149.201.147:4021 , piece_id=Z6PYBMORPCESZXX3CUAJRANAJ64LTCO742BKUN3F2CBLZO3IC7DA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.112701080 (44h2m11.583911819s) [noise=true , address=89.149.201.147:4024 , piece_id=FBE2CRSK46LOBYES62HZEIJD6UGK7XJMJP24J6XLAGZUKNEK2ZKQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:57:35.148224273 (44h2m11.619435002s) [noise=true , address=89.149.201.147:4018 , piece_id=WBLQVRRTKGEJJ75ZPWN7CXK65K5ZXOVH6QTIGY34RL35EZS3G73Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.601076549 (44h2m43.072287288s) [noise=true , address=185.167.204.206:4072 , piece_id=GTEJKPFUP4OMHHPMGGNUYAOTRRPSKA7MNPRN5UANZCUCRUJCZQ6A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.858730070 (44h2m43.329940809s) [noise=true , address=185.167.204.205:4013 , piece_id=AWU4445M6HGFROHD7ZON7LF4XHHDBWNGYQ5OBTWRT6ZG7NKIQYPQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.984091582 (44h2m43.455302321s) [noise=true , address=185.167.204.205:4026 , piece_id=4QUQNQEML7X3MWDT3BW2V6HVKMEB34HY5ZOCGJTD4LDH7TOABC4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:06.997908141 (44h2m43.46911888s) [noise=true , address=185.167.204.205:4027 , piece_id=XE55BFKL5EEYCUI6F3CG2OMHWXTVPADE6N2BIEFODT77MVXUPTKA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:07.704382372 (44h2m44.175593111s) [noise=true , address=185.167.204.206:4052 , piece_id=JV4GC5Y3HXU44FVMMZZDHMN5LD6AK4SO3236KBKKZEZJQ3VOMHLQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:08.306751277 (44h2m44.777962016s) [noise=true , address=185.167.204.205:4032 , piece_id=IL72GNJOSYSISBLOZZXISW6BNHXNSUWMBTKBBWCRI4EWG5WSHTZA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:09.728807001 (44h2m46.20001773s) [noise=true , address=185.167.204.205:4001 , piece_id=M6RNATECMXPRLAIBWCYRLCOCEBZHGTZAM2PFPEUGGB2LXT3TKW4Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:10.200462412 (44h2m46.671673151s) [noise=true , address=185.167.204.206:4043 , piece_id=ENHMBBAPVWOYLWG6S2AB75MWWY65ZMWVFGT6LQR6OOH6AMBAANGA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:11.218612754 (44h2m47.689823493s) [noise=true , address=185.167.204.205:4030 , piece_id=JBKFARJUDMQFJIUND5IQTHLCNHQVYHR4P5XOQK24RCDCERC5X72A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:22.049936661 (44h2m58.52114739s) [noise=true , address=185.167.204.205:4014 , piece_id=SZPBSMVDNUZ5IT4O7RFMEHY5EY64L2XO545LU6IRO6HFDSUYINXQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:23.835508896 (44h3m0.306719625s) [noise=true , address=185.167.204.206:4056 , piece_id=GEVQRJW4TE7XTCGJX5HVUSBIKML5ESGM5NWVVDU7ELANJRRT4W5Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:26.013124099 (44h3m2.484334839s) [noise=true , address=185.167.204.206:4041 , piece_id=TOGP35PMCLIUVYB4J6N64ARHRMHKVKQE2IONNQEW32KCPNQYPL2A, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:27.982649379 (44h3m4.453860109s) [noise=true , address=185.167.204.205:4021 , piece_id=D6JKU2MXC3JPLXAGMZCG4BS7HQ3FCQ4XXBSNEAHMVYKGG66SSTQQ, seg_pos={0 0}, upload=38726]: Done uploading piece. err: <nil>
Dec 5 12:58:31.996355435 (44h3m8.467566194s) [noise=true , address=185.167.204.205:4023 , piece_id=ILTPO5YLTESQGUBGOJXI4VSRW4VTWW5CYYNHWVVYWBOM6ULD2UBA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:12Mr4R5VQs9NbCHNZonSC42mesJNuJ2Co3BbU7LTfLPQwWEkQbo, address:185.167.204.205:4023): protocol: write tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; close tcp 188.166.225.170:34390->185.167.204.205:4023: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996542821 (44h3m8.46775356s) [noise=true , address=185.167.204.205:4035 , piece_id=MGAKVZ4LYIJ27QPPLUJK5CHPALW2P2FCQF2SHHP6SB7NKD2PUS2Q, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996709960 (44h3m8.467920699s) [noise=true , address=185.167.204.205:4009 , piece_id=HVAWJST2LC3GRHBQKGMDBQNUJLZY6MWNXZWNKFT5BBMYDK7TZEWA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:1bGUwUMQeVuL4HCvsRkMSPsgzByNWwMvx6t7HmQ7WLsXoJytgb, address:185.167.204.205:4009): protocol: rpc: dial tcp 185.167.204.205:4009: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded
Dec 5 12:58:31.996823235 (44h3m8.468033974s) [noise=true , address=185.167.204.205:4002 , piece_id=4CRJ6ZV5WOJVT7WCEGZHU5HQ56VXZXJ4RB4IBLTRC6CPUQTFOLRA, seg_pos={0 0}, upload=38726]: Done uploading piece. err: ecclient: upload failed (node:122KLY1enErmKthxbRteifu9eqBKwRCwyvmFdM1yErYQnXQpxQ3, address:185.167.204.205:4002): protocol: rpc: dial tcp 185.167.204.205:4002: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 5 12:58:31.996866238 (44h3m8.468076977s) [seg_pos={0 0}, upload=38726]: Done waiting for segment. successful: 24 optimal: 26 errs: <nil>
Dec 5 12:58:32.172198764 (44h3m8.643409493s) [upload=38726]: Done uploading segments...
Dec 5 12:58:32.172234372 (44h3m8.643445101s) [upload=38726]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 24)
Duration: 5.786503341s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.434919438s trace:
https://jaeger.dev.storj.io/trace/63fbf6b6cd5d03cb
FAIL (2025-12-05 13:39:02 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 51)
Duration: 5.434955246s
core
uplink
eu1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
6.103599958s trace:
https://jaeger.dev.storj.io/trace/275a4a7f39135e2a
FAIL (2025-12-05 13:38:55 UTC)
orbiter warning: operation: Commit uplink: failed to upload enough pieces (needed at least 54 but got 45)
Duration: 6.103625917s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [X...XXX.XX..X.XXX.XXXXXXXXX.X..XXX..X..XXX..XXXX.XXXX.XXX.XX] duration:
5.001748522s trace:
https://jaeger.dev.storj.io/trace/65a68dc08f702adb
FAIL (2025-12-05 13:38:52 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001774333s
core
uplink
us-select-1
us1-placement-20
|