Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
5.721896519s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:20:52 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.721935373s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.138720777s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:19:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.138753511s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.027906662s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:19:04 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.028016332s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.301188575s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:18:33 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.30121747s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.479838229s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:18:16 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.479929753s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.750641047s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:17:21 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.75067472s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
5.918752959s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:16:54 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.918783979s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
5.823174292s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:15:57 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.823208837s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
6.100312771s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:15:17 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.100338259s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
5.57679538s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:14:43 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.576825427s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.863863384s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:14:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.863890906s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.000913546s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:13:24 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.000944906s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.21141784s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:12:19 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.211467015s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
5.64716472s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:12:01 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.647189919s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.757797626s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:11:45 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.757816873s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
5.892053023s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:11:30 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.892082129s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
6.012745848s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:11:14 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.01277362s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
5.78816457s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:10:49 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.788190249s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
5.960730122s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:10:09 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.960768455s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.584899767s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:09:44 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.584927299s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
7.701699414s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:09:17 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 7.701723349s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.841590395s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:08:19 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.841617718s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.953009417s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:07:28 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.95302661s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
5.95857788s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:06:34 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.958659044s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.329127819s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:05:32 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.329293836s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.922616524s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:05:11 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.922644647s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
5.774664383s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:04:06 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.774690253s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.518772707s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:03:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.518799779s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
5.899966109s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:03:10 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.900100213s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.095965653s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:02:10 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 6.0959957s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.980543646s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:01:44 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.980578041s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.691542975s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:01:24 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.691559898s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.850270703s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 07:00:23 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.850303767s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.7323717s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 06:59:39 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.732402959s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
5.933740434s trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 06:58:51 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 5.933783266s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
458.685033ms trace:
https://jaeger.dev.storj.io/trace/4d3274db26fab217
FAIL (2025-12-12 06:57:55 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:59068->45.38.24.2:443: read: connection reset by peer
Duration: 458.711324ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
8.312424284s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:38:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 8.312502845s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
7.22550529s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:37:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 7.22553642s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
7.093845058s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:37:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 7.093881257s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.02460259s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:36:30 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.024634841s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.72194913s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:36:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 6.721977424s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.185653604s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:35:48 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.185718438s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
7.119162701s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:35:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 7.11918806s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.781052386s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:35:11 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.781087782s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.903794396s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:35:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 6.903829423s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.327965874s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:34:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 6.32799555s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.942871798s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:34:24 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.942900303s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.392323992s trace:
https://jaeger.dev.storj.io/trace/1feedb9fc9b15c7f
FAIL (2025-12-12 06:33:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP, Piece ID: AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ) context deadline exceeded; piecestore: (Node ID: 1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, Piece ID: XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ) context deadline exceeded; piecestore: (Node ID: 12HWX3mC14qJg7WdCetaBtS2GcGQFhzSaqUbKrYFBHZKdyFAAgX, Piece ID: ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q) context deadline exceeded; piecestore: (Node ID: 12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, Piece ID: T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A) context deadline exceeded; piecestore: (Node ID: 1uzGTks5tffchQBGQK23fnWZXMfrsTHh4Zqs9UGwnWUmnQghEW, Piece ID: V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA) context deadline exceeded
======LOGS======
Dec 12 05:52:28.758186788 (158h23m28.629269833s) [upload=138545]: Starting upload
Dec 12 05:52:28.758268054 (158h23m28.629351089s) [upload=138545]: Done starting upload
Dec 12 05:52:28.758478524 (158h23m28.62956157s) [upload=138545]: Uploading segments...
Dec 12 05:52:28.758589055 (158h23m28.62967209s) [upload=138545]: Got next segment. Inline: false
Dec 12 05:52:28.758627448 (158h23m28.629710493s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 12 05:52:28.758633229 (158h23m28.629716264s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 12 05:52:29.284642531 (158h23m29.155725586s) [seg_pos={0 0}, upload=138545]: Begin upload segment...
Dec 12 05:52:29.284698698 (158h23m29.155781743s) [seg_pos={0 0}, upload=138545]: Done begin upload segment.
Dec 12 05:52:29.284747200 (158h23m29.155830245s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.285714762 (158h23m29.156797808s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286167084 (158h23m29.157250119s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.286680902 (158h23m29.157763937s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.287180654 (158h23m29.158263699s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288071100 (158h23m29.159154145s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.288112027 (158h23m29.159195083s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289064060 (158h23m29.160147095s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.289119456 (158h23m29.160202501s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290114690 (158h23m29.161197735s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.290153143 (158h23m29.161236189s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291232188 (158h23m29.162315233s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291345174 (158h23m29.162428219s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.291791403 (158h23m29.162874458s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292096845 (158h23m29.16317988s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:29.292469154 (158h23m29.163552199s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Uploading piece...
Dec 12 05:52:31.055397931 (158h23m30.926480977s) [upload=138545]: Next returned nil segment
Dec 12 05:52:31.055414492 (158h23m30.926497538s) [upload=138545]: Waiting for error group managing segments...
Dec 12 05:52:31.488724096 (158h23m31.359807131s) [noise=true , address=77.90.31.18:4805 , piece_id=256JZ54CU4FB2R7TGIAMUUWWLZYHQKFB3HYEU7IXB2L34XBCN5FA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.526825088 (158h23m31.397908123s) [noise=true , address=77.90.31.16:4735 , piece_id=Q55QMXTDMSRAIU3NVRA77P3WXRUGUVKPUFSLOP4B5MU67S7OUR2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.568810436 (158h23m31.439893481s) [noise=true , address=145.79.227.36:4517 , piece_id=T2O7DFZXJ6TEFMX75K5JK7X5PJR5GGPY65DHJDKYRZIHGZI46W2A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.593069069 (158h23m31.464152114s) [noise=true , address=145.79.227.34:4905 , piece_id=OCJKQDITIBYRSSUTSVWJYCOVEKMMHT5FBPFIK3ZPITWHN6O6HN5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.684591374 (158h23m31.555674419s) [noise=true , address=145.79.227.49:4337 , piece_id=67URJKLDGXQEMUL4R5FFY4UYLUPQCTIAV2IUDOYMCSAPDIMGYA7A, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:31.981164324 (158h23m31.852247369s) [noise=true , address=145.79.227.19:4078 , piece_id=XXQDBSEQC2JH2UDIPGIUL35OOONZTK6TACRVTQRNR4X5TADMFJOQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.086675528 (158h23m31.957758573s) [noise=true , address=145.79.227.27:4689 , piece_id=PSZIMT7PXTRLC6QQSHMR5IO2WQQBBB62DNZPQQCB5GCM2GYG76PA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:32.328097079 (158h23m32.199180124s) [noise=true , address=145.79.227.52:4447 , piece_id=AGN7AA5B5BVFSTKCS3TN75K2AO7RANBXMJFFLLUGFLBC5A54NNBQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.363427452 (158h23m33.234510497s) [noise=true , address=145.79.227.47:4266 , piece_id=V6FYPFYCCQTS3GULHYFZOY6JDAVKHNMQ4I77C6KFWSFYZRGBUASA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.581309902 (158h23m33.452392947s) [noise=true , address=145.79.227.46:4601 , piece_id=6F2OKCLBRK3JRME7NOEUJ4TEIVIOX3CJGKCVGROCHVZ6GDW3KYPA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:33.597965646 (158h23m33.469048692s) [noise=true , address=145.79.227.32:4836 , piece_id=IZWT5CMWWQFF2MVLK7RWQ3TJ7SGMILNNRTYOB6OMNQ5VWVT42KDQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.127692344 (158h23m34.998775389s) [noise=true , address=145.79.227.51:4410 , piece_id=ZE4XSENLNTPSLNUHRBCSFIKEB3FDZCVMD3TLJJSQDG6AGASJAW5Q, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520638237 (158h23m35.391721292s) [noise=true , address=145.79.227.4:4185 , piece_id=HKTBVHTFWTNTAVGRNZTFQRZZHNBSSWNHEXEXWGUBZAWUM2OHTRLQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: <nil>
Dec 12 05:52:35.520664637 (158h23m35.391747712s) [seg_pos={0 0}, upload=138545]: Segment reached optimal threshold of 13 pieces.
Dec 12 05:52:35.521193143 (158h23m35.392276239s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bUyv2E7SXv5BCBjxbjHp99e4dKkuEwZpNv9zbFqddhD11CTq8): protocol: read tcp 188.166.225.170:56866->145.79.227.25:4618: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521205126 (158h23m35.392288181s) [noise=true , address=145.79.227.25:4618 , piece_id=M47MFLF7ZOWMCGD7K7WIHA6I3RFLI4MI7TSUWNH7WMTEYD2AC7ZQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.521775052 (158h23m35.392858107s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: read tcp 188.166.225.170:35500->145.79.227.43:4159: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 12 05:52:35.521788437 (158h23m35.392871472s) [noise=true , address=145.79.227.43:4159 , piece_id=WNTFV3DGNXSBDQ74W6WKY7S2BPFRLMXT5LOC2DMJIR6ZOPCPTKQA, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522125940 (158h23m35.393208985s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17x7QtnZsENMZPU3tqkh8jEd29JdowLvT855W4kuourgUA6y5C): protocol: expected piece hash; context canceled; EOF
Dec 12 05:52:35.522137161 (158h23m35.393220206s) [noise=true , address=145.79.227.41:4085 , piece_id=2NAFYOEP5YOQRA6LJGOX24EXNNGIL2NPR6WKETE5WJWVSILDHJJQ, seg_pos={0 0}, upload=138545]: Long tail cancellation due to optimal threshold
Dec 12 05:52:35.522160766 (158h23m35.393243801s) [seg_pos={0 0}, upload=138545]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 12 05:52:35.522242732 (158h23m35.393325777s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 12 05:52:35.522248944 (158h23m35.393331989s) [upload=138545]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 12 05:52:35.844205119 (158h23m35.715288164s) [upload=138545]: Done uploading segments...
Dec 12 05:52:35.844231129 (158h23m35.715314164s) [upload=138545]: Upload finished. err: <nil>
Duration: 6.392351226s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.718923843s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:33:12 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.718956315s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.936328839s trace:
https://jaeger.dev.storj.io/trace/76c0b0a8c78b2475
FAIL (2025-12-12 06:32:54 UTC)
orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.936357283s
core
uplink
us1
|