Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [................................X...........................] duration:
3.38467065s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:12:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.384697694s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................X....................X.............] duration:
5.636929891s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:12:47 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.636948437s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......................X.X......X.................X..........] duration:
5.867824982s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:12:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.867854889s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.................................X..........................] duration:
3.521464942s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:12:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.521521089s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................X....................X............] duration:
6.645196207s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:11:44 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.645228789s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......................X.X......X.................X.........] duration:
5.653612455s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:11:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.653643864s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........................X....................X...........] duration:
5.631976569s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:11:05 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.632010142s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [X.................................X.........................] duration:
3.455090192s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:10:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.455123758s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........................X.X......X.................X........] duration:
5.90135612s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:10:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.90138804s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.X.................................X........................] duration:
3.496136202s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:10:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.496190996s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............................X....................X..........] duration:
6.119700641s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:10:13 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.119729556s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........................X.X......X.................X.......] duration:
5.600900773s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:09:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.600930449s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............................X....................X.........] duration:
6.039210301s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:09:25 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.03923567s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..X.................................X.......................] duration:
3.459596768s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:09:13 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.459628718s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........................X.X......X.................X......] duration:
5.845473234s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:08:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.845535373s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..............................X....................X........] duration:
5.744471409s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:08:16 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.744507038s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...X.................................X......................] duration:
3.478396463s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:08:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.478573049s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...........................X.X......X.................X.....] duration:
5.689342826s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:08:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.689382932s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X..............................X....................X.......] duration:
6.96514978s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:07:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 6.965178304s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X...........................X.X......X.................X....] duration:
5.739258334s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:07:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.739283742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....X.................................X.....................] duration:
3.519974645s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:06:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.519997077s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.X...........................X.X......X.................X...] duration:
5.796306554s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:06:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.796335349s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X..............................X....................X......] duration:
5.83314992s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:06:21 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.833181159s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..X...........................X.X......X.................X..] duration:
5.738245284s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:06:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.738277305s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X..............................X....................X.....] duration:
7.306160409s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:06:06 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 7.306192088s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.....X.................................X....................] duration:
3.363927736s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:05:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.363954476s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X...........................X.X......X.................X.] duration:
6.590217305s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:05:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.590252551s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [....X...........................X.X......X.................X] duration:
1m0.001596268s trace:
https://jaeger.dev.storj.io/trace/5206c43f0c9eadf2
FAIL (2025-12-08 07:05:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.001619542s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X..............................X....................X....] duration:
5.708882341s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:05:14 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.708909703s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [......X.................................X...................] duration:
3.549876302s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:05:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.549982024s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X..............................X....................X...] duration:
5.752475982s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:04:53 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.752508474s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......X.................................X..................] duration:
4.308185214s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:04:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.308283902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X..............................X....................X..] duration:
5.953334785s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:04:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.953365975s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.....X...........................X.X......X.................] duration:
5.991032751s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:03:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.991059391s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [........X.................................X.................] duration:
3.46031409s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:03:36 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.460344037s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X..............................X....................X.] duration:
5.538283198s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:03:30 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 5.538304128s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X..............................X....................X] duration:
3m54.084459253s trace:
https://jaeger.dev.storj.io/trace/10deaed4f0583195
FAIL (2025-12-08 07:03:24 UTC)
orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded
Duration: 3m54.084483568s
core
uplink
us1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [......X...........................X.X......X................] duration:
5.656200154s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:03:21 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.656231464s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.........X.................................X................] duration:
3.589824956s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:03:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.589856736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.......X...........................X.X......X...............] duration:
5.777288972s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:02:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.777357562s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..........X.................................X...............] duration:
3.420065279s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:02:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.420093453s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [........X...........................X.X......X..............] duration:
5.911795797s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:02:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.911829502s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...........X.................................X..............] duration:
4.331082612s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:01:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.33113359s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [............X.................................X.............] duration:
3.408931156s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:01:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.409170491s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.458754443s trace:
https://jaeger.dev.storj.io/trace/2313adc163c8e057
FAIL (2025-12-08 07:01:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload
Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload
Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments...
Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false
Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment...
Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment.
Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000
Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment
Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments...
Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces.
Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF
Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection
Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments...
Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.458782096s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.........X...........................X.X......X.............] duration:
5.849790999s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:01:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.84981768s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.218919868s trace:
https://jaeger.dev.storj.io/trace/2313adc163c8e057
FAIL (2025-12-08 07:01:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload
Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload
Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments...
Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false
Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment...
Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment.
Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000
Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment
Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments...
Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces.
Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF
Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection
Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments...
Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.21894658s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
6.441313645s trace:
https://jaeger.dev.storj.io/trace/2313adc163c8e057
FAIL (2025-12-08 07:00:50 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 8 06:09:37.072171265 (62h40m36.9432543s) [upload=54958]: Starting upload
Dec 8 06:09:37.072229475 (62h40m36.94331251s) [upload=54958]: Done starting upload
Dec 8 06:09:37.072409157 (62h40m36.943492202s) [upload=54958]: Uploading segments...
Dec 8 06:09:37.072514628 (62h40m36.943597673s) [upload=54958]: Got next segment. Inline: false
Dec 8 06:09:37.072727844 (62h40m36.943810889s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 8 06:09:37.072736801 (62h40m36.943819836s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 8 06:09:38.737650041 (62h40m38.608733076s) [seg_pos={0 0}, upload=54958]: Begin upload segment...
Dec 8 06:09:38.737704164 (62h40m38.608787209s) [seg_pos={0 0}, upload=54958]: Done begin upload segment.
Dec 8 06:09:38.737740112 (62h40m38.608823157s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.737870721 (62h40m38.608953756s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738399147 (62h40m38.609482182s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.738696634 (62h40m38.609779669s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739054575 (62h40m38.61013761s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739369375 (62h40m38.61045242s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.739704062 (62h40m38.610787097s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740023320 (62h40m38.611106365s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.740440524 (62h40m38.61152357s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.741692268 (62h40m38.612775313s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742904676 (62h40m38.613987711s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.742983607 (62h40m38.614066642s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.744313870 (62h40m38.615396915s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.745616059 (62h40m38.616699094s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.747217098 (62h40m38.618300133s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:38.748821062 (62h40m38.619904097s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Uploading piece...
Dec 8 06:09:40.140688699 (62h40m40.011771764s) [noise=true , address=77.90.31.11:4553 , piece_id=YJ6NPRPQ3NDQGHGTNIIV7WIYBERS6UT2GTXOREEEPTVPRMSSBWOA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: storage node overloaded, request limit: 2000
Dec 8 06:09:40.475776076 (62h40m40.346859112s) [upload=54958]: Next returned nil segment
Dec 8 06:09:40.475790184 (62h40m40.346873219s) [upload=54958]: Waiting for error group managing segments...
Dec 8 06:09:40.760659949 (62h40m40.631742984s) [noise=true , address=145.79.227.16:4911 , piece_id=YUO2CK7GGLTY2XWEKFHO75NO7FE62SFH63BCQ4USBEVP5ACXULIA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.783210068 (62h40m40.654293103s) [noise=true , address=145.79.227.39:4014 , piece_id=FMNJXPUL2IL4V76MIXZADIXLO2NGFA6N25AJ4FQVIYV3ZJB3CZXA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.798552993 (62h40m40.669636028s) [noise=true , address=145.79.227.28:4726 , piece_id=WFMID44RDZNHCB4ZJIFU34AUCV4A44HLDLJWWFYTLTKH22I4CQPA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.806464277 (62h40m40.677547312s) [noise=true , address=77.90.31.18:4807 , piece_id=XCF3XHLDM2WS5AIXMLLTBPXXCEUYNQYZUKNLMTXW3TINTZM3YOHA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.847817199 (62h40m40.718900245s) [noise=true , address=77.90.31.10:4518 , piece_id=3ON6IZXND2E2G7IHYBZA67Z2UWSW5MVKMSHXHTPFOFQ4PNN46K6A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.870832625 (62h40m40.74191566s) [noise=true , address=145.79.227.39:4015 , piece_id=VTZXPQ6XLTYV72SWOSKEISUJ5WA7T2T3X4JHLRXEAGUX5UTHPU5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.875162790 (62h40m40.746245845s) [noise=true , address=145.79.227.32:4834 , piece_id=4C5LOK75SET35U6YTRKNTPNEWGLJC5MOREWHMXWMYVGWDXAWXLSA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.894092305 (62h40m40.76517535s) [noise=true , address=145.79.227.25:4617 , piece_id=O3PRED26EI3VKRCE7ZCPK6I7D3TMLBB74TERCW4JLWNQCM67ZJZQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:40.919907982 (62h40m40.790991017s) [noise=true , address=145.79.227.26:4655 , piece_id=WWRKIUNVY3MTOU2EAVVXP7ZJYIFYGWTKNPXICSL7DKSUQAQESNEA, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.920978983 (62h40m41.792062028s) [noise=true , address=145.79.227.22:4511 , piece_id=SHH42FTBMITTZHPH5T5IKGLAORB4T3V3HO5B4MZQY5C4XTZKTW7Q, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:41.954976078 (62h40m41.826059133s) [noise=true , address=145.79.227.38:4979 , piece_id=LIFD2YFO6QNTUHLWNDE47LRQUDLWSOPRR7J7YICUNDJJPDWROMVQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.012789744 (62h40m41.88387279s) [noise=true , address=145.79.227.11:4730 , piece_id=HZL5I3IVLULMPRBXP4CJTNJWL6ASVGUPIQGQ57K2OOS6DEAIQJ7A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013556745 (62h40m41.88463979s) [noise=true , address=145.79.227.43:4157 , piece_id=F26DWJJJRT6EKBFOWLBOKERS7DHPNBPAEPRIVC5WWTJ4F7GOESIQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: <nil>
Dec 8 06:09:42.013571063 (62h40m41.884654098s) [seg_pos={0 0}, upload=54958]: Segment reached optimal threshold of 13 pieces.
Dec 8 06:09:42.013862497 (62h40m41.884945552s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YKLTah3swX14kL82eNogJxd8D8Hjfq7SPjDDQAapDnXmDctP1): protocol: expected piece hash; context canceled; EOF
Dec 8 06:09:42.013874110 (62h40m41.884957155s) [noise=true , address=145.79.227.28:4725 , piece_id=TX4VZH77P5SYUKC2ZEXPOJZDK4CZEMSGS4PEI7LNQRIVZ424WA5A, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013943172 (62h40m41.885026207s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55090->77.90.31.12:4590: use of closed network connection
Dec 8 06:09:42.013951728 (62h40m41.885034783s) [noise=true , address=77.90.31.12:4590 , piece_id=WGHUFVWOQ2AP5RFK4NUK3DVJDGYCBZPCCBPMAF7B4CZ5UWSPFUPQ, seg_pos={0 0}, upload=54958]: Long tail cancellation due to optimal threshold
Dec 8 06:09:42.013993407 (62h40m41.885076442s) [seg_pos={0 0}, upload=54958]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 8 06:09:42.014089720 (62h40m41.885172766s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 8 06:09:42.014097605 (62h40m41.88518065s) [upload=54958]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 8 06:17:18.682530512 (62h48m18.553613557s) [upload=54958]: Done uploading segments...
Dec 8 06:17:18.682616375 (62h48m18.55369941s) [upload=54958]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.441346768s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..........X...........................X.X......X............] duration:
5.897275749s trace:
https://jaeger.dev.storj.io/trace/7aa9251ce4234ecb
FAIL (2025-12-08 07:00:32 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.897363187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.............X.................................X............] duration:
3.435670965s trace:
https://jaeger.dev.storj.io/trace/7bdbfcfb6fddc4a0
FAIL (2025-12-08 07:00:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.435706484s
core
uplink
us-select-1
us1-placement-20
|