Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.................X.X.X...........XXXX....XXX..X...XX........] duration:
6.99639088s
FAIL (2025-09-13 16:05:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.996414726s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..................X.X.X...........XXXX....XXX..X...XX.......] duration:
6.588878475s
FAIL (2025-09-13 16:05:10 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.588908001s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [...................X.X.X...........XXXX....XXX..X...XX......] duration:
6.920894505s
FAIL (2025-09-13 16:04:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.920919955s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [....................X.X.X...........XXXX....XXX..X...XX.....] duration:
7.247679952s
FAIL (2025-09-13 16:04:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.247716833s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.....................X.X.X...........XXXX....XXX..X...XX....] duration:
7.408827561s
FAIL (2025-09-13 16:03:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.408872918s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [......................X.X.X...........XXXX....XXX..X...XX...] duration:
7.209606479s
FAIL (2025-09-13 16:03:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.209631217s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.......................X.X.X...........XXXX....XXX..X...XX..] duration:
8.731213526s
FAIL (2025-09-13 16:02:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.731236882s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [........................X.X.X...........XXXX....XXX..X...XX.] duration:
7.798761404s
FAIL (2025-09-13 16:01:49 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.798799066s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.........................X.X.X...........XXXX....XXX..X...XX] duration:
30.001433232s
FAIL (2025-09-13 16:01:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001454863s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..........................X.X.X...........XXXX....XXX..X...X] duration:
30.001039113s
FAIL (2025-09-13 16:00:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001058119s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...........................X.X.X...........XXXX....XXX..X...] duration:
7.367428216s
FAIL (2025-09-13 15:59:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.367448485s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [............................X.X.X...........XXXX....XXX..X..] duration:
7.222587988s
FAIL (2025-09-13 15:58:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.222605742s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X............................X.X.X...........XXXX....XXX..X.] duration:
7.134894301s
FAIL (2025-09-13 15:58:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.134907125s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X............................X.X.X...........XXXX....XXX..X] duration:
30.00192938s
FAIL (2025-09-13 15:57:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001963217s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [X.X............................X.X.X...........XXXX....XXX..] duration:
6.878612894s
FAIL (2025-09-13 15:57:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.878636067s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [.X.X............................X.X.X...........XXXX....XXX.] duration:
6.707098155s
FAIL (2025-09-13 15:56:17 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.707131338s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..X.X............................X.X.X...........XXXX....XXX] duration:
30.001130379s
FAIL (2025-09-13 15:55:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001154505s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...X.X............................X.X.X...........XXXX....XX] duration:
30.000673955s
FAIL (2025-09-13 15:54:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000686499s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X...X.X............................X.X.X...........XXXX....X] duration:
30.000826299s
FAIL (2025-09-13 15:54:04 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000848231s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X...X.X............................X.X.X...........XXXX....] duration:
7.237802728s
FAIL (2025-09-13 15:52:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.237833327s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X.X...X.X............................X.X.X...........XXXX...] duration:
7.211197181s
FAIL (2025-09-13 15:52:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.211221007s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [XX.X...X.X............................X.X.X...........XXXX..] duration:
7.324716163s
FAIL (2025-09-13 15:51:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.324739528s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [XXX.X...X.X............................X.X.X...........XXXX.] duration:
7.309091458s
FAIL (2025-09-13 15:50:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.309117918s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [.XXX.X...X.X............................X.X.X...........XXXX] duration:
30.001159396s
FAIL (2025-09-13 15:50:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001183002s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [..XXX.X...X.X............................X.X.X...........XXX] duration:
30.000730359s
FAIL (2025-09-13 15:48:54 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000751571s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...XXX.X...X.X............................X.X.X...........XX] duration:
30.001993756s
FAIL (2025-09-13 15:48:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.002008372s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [....XXX.X...X.X............................X.X.X...........X] duration:
30.000760282s
FAIL (2025-09-13 15:47:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000784548s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....XXX.X...X.X............................X.X.X...........] duration:
7.263709151s
FAIL (2025-09-13 15:46:55 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.263723127s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [......XXX.X...X.X............................X.X.X..........] duration:
7.448043027s
FAIL (2025-09-13 15:46:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.44806561s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......XXX.X...X.X............................X.X.X.........] duration:
7.327243582s
FAIL (2025-09-13 15:45:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.32726861s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [........XXX.X...X.X............................X.X.X........] duration:
6.967673107s
FAIL (2025-09-13 15:45:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.967712039s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
23.603821193s
FAIL (2025-09-13 15:45:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 23.603849577s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........XXX.X...X.X............................X.X.X.......] duration:
6.654297506s
FAIL (2025-09-13 15:44:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.654322093s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
18.143162431s
FAIL (2025-09-13 15:43:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 18.143182129s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X.........XXX.X...X.X............................X.X.X......] duration:
7.201472524s
FAIL (2025-09-13 15:43:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.201495549s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
18.10743396s
FAIL (2025-09-13 15:42:37 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 18.107457525s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X.........XXX.X...X.X............................X.X.X.....] duration:
7.082947902s
FAIL (2025-09-13 15:42:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.082968962s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
19.59990458s
FAIL (2025-09-13 15:42:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 19.59993066s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [X.X.........XXX.X...X.X............................X.X.X....] duration:
7.137760534s
FAIL (2025-09-13 15:41:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.137776715s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [.X.X.........XXX.X...X.X............................X.X.X...] duration:
7.003893314s
FAIL (2025-09-13 15:41:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.003916048s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [X...X.......................................................] duration:
16.87606064s
FAIL (2025-09-13 15:41:14 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 16.876078545s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
8.91022139s
FAIL (2025-09-13 15:41:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.910244964s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [..X.X.........XXX.X...X.X............................X.X.X..] duration:
7.784898698s
FAIL (2025-09-13 15:40:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.785083531s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
9.36242675s
FAIL (2025-09-13 15:40:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.362448401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [.X...X......................................................] duration:
20.94231225s
FAIL (2025-09-13 15:40:06 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 20.942336606s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
8.455631379s
FAIL (2025-09-13 15:39:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.455643963s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [...X.X.........XXX.X...X.X............................X.X.X.] duration:
9.49005925s
FAIL (2025-09-13 15:39:35 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 9.490071474s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....X.X.........XXX.X...X.X............................X.X.X] duration:
30.000743483s
FAIL (2025-09-13 15:39:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000765766s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration:
17.56506752s
FAIL (2025-09-13 15:38:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 3)
======LOGS======
6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.907845810 (169h26m51.94741893s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908062555 (169h26m51.947635675s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908276674 (169h26m51.947849794s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:15.908522154 (169h26m51.948095274s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Uploading piece...
Sep 13 14:55:17.522666509 (169h26m53.562239629s) [upload=148985]: Next returned nil segment
Sep 13 14:55:17.522686247 (169h26m53.562259367s) [upload=148985]: Waiting for error group managing segments...
Sep 13 14:55:18.048737965 (169h26m54.088311085s) [noise=true , address=77.90.31.17:4793 , piece_id=PWVTHYWQY6GFO5FCIBANYNSX72VNUGT4UXKYI5CEIIHNZGNB63OA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.323717226 (169h26m54.363290346s) [noise=true , address=77.90.31.11:4559 , piece_id=OYMOQVJRZ6SMMF7O2DXXDEOVRN27B3FT5EIVMFFBHC47UPZUUMIQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:55:18.769215263 (169h26m54.808788383s) [noise=true , address=45.91.116.14:4221 , piece_id=Q7ZPEITLUERUPOI247KAARIL3GSAPW6JNEUF7FSHQWJHL5GTGIZQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: <nil>
Sep 13 14:56:14.598217910 (169h27m50.63779103s) [noise=true , address=145.79.227.28:4730 , piece_id=5WCQINYSLLWZR3Z3QWWUKRKDPRWYQ6VQPIZ4YDAD47JHSKDSCAGA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1SXQt5FugmWj1YdhsvaaR7bDoxTBkqHKZpyXSmQqGdX5ZRhKxV, address:145.79.227.28:4730): protocol: read tcp 188.166.225.170:42820->145.79.227.28:4730: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598245002 (169h27m50.637818122s) [noise=true , address=145.79.227.52:4475 , piece_id=3T3MRZJ2T74J7CB5EJ4N2CIIERKWTQ5KWIOLDNV2LXHYLO5IKHTA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12UYjE6PEB9gZKwTKcnx23ykF4jDMJzLQoKmiFiejWew5mirdCj, address:145.79.227.52:4475): protocol: rpc: dial tcp 145.79.227.52:4475: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598273767 (169h27m50.637846887s) [noise=true , address=145.79.227.50:4386 , piece_id=74QVIDRRHFRIA5Z6ZAQ4EWQ2PZ5HZI5JXNJ5WR2DVFJQ4WGE2UPQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:19aPjJnya2Q3stEhiedDSth3NHV8phLRj5yY79hZv81UgDitMv, address:145.79.227.50:4386): protocol: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42008->145.79.227.50:4386: use of closed network connection
Sep 13 14:56:14.598293245 (169h27m50.637866365s) [noise=true , address=145.79.227.5:4250 , piece_id=EPZOZXRFVPSAPD6UKPOCCBOX7VJNB4DQRKV6YTRWMPOLQQ5IH6KQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1uoGjfqBLfoNrg2X1aYKYLKQjgFUnzLjHetCFc9sPjJa2qqKYM, address:145.79.227.5:4250): protocol: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58078->145.79.227.5:4250: use of closed network connection
Sep 13 14:56:14.598308794 (169h27m50.637881914s) [noise=true , address=145.79.227.33:4894 , piece_id=5KD2BL5J7DT7JDLPDS2QUUMM26AFWSXJ2BDTJSUGZ253ZV7XOVKQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12KWNyruYhmSJVvgUF7fBpGf4H7eYmtAeEXToYjd2dcEPcj1E44, address:145.79.227.33:4894): protocol: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53614->145.79.227.33:4894: use of closed network connection
Sep 13 14:56:14.598327330 (169h27m50.63790045s) [noise=true , address=145.79.227.7:4308 , piece_id=FYL7PQOKW74EJ2QLKMN6EYJKIT7A6B2VQESVP2VFVDVJHPHXFGAQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37146->145.79.227.7:4308: use of closed network connection
Sep 13 14:56:14.598358169 (169h27m50.637931289s) [noise=true , address=145.79.227.36:4544 , piece_id=WZRHCCBJDHID4R4UOB76OABJSH2QVC6WZ6OZWIUUMKWWBDVWEG4A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12RSxwxw1WXkNcu7b3dEgJhyNkFGYUfKF4QVKmK323L5g3EKFnG, address:145.79.227.36:4544): protocol: rpc: dial tcp 145.79.227.36:4544: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598387545 (169h27m50.637960665s) [noise=true , address=145.79.227.2:4137 , piece_id=NH4RZXYD3LKQZLLP5SSA7HXC7FBBOQV6QLBE2B3R5HH5F23Q7E6A, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:145.79.227.2:4137): protocol: rpc: dial tcp 145.79.227.2:4137: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598421721 (169h27m50.637994841s) [noise=true , address=145.79.227.30:4780 , piece_id=7EMNYDLOMVLIDGE5ZK7WFQCNGBFLZA5AF5EHSLHYE22XJ2MP56VQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:145.79.227.30:4780): protocol: read tcp 188.166.225.170:43016->145.79.227.30:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598427161 (169h27m50.638000281s) [noise=true , address=145.79.227.19:4090 , piece_id=YZ2H4UUM3Z5ZMCC6LQDAPLHCT7OQQAMRSDBQ3CH6I6UT2HAN4TXQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12qSkJk9HUyQon3UP1AA8mRcqJbNdkKJh6cEUZoHLGNniqDvNqB, address:145.79.227.19:4090): protocol: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37110->145.79.227.19:4090: use of closed network connection
Sep 13 14:56:14.598456818 (169h27m50.638029938s) [noise=true , address=145.79.227.10:4425 , piece_id=CRES4FQPX4WMTDTUBCPRZ2R4OHTI7MMHS6XSCZ7UXKDLSSLEVQ3Q, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:13sAcGFjAXuesAkUn7NuLaaV8TAah3Mr1kqd33wwh9MTELqmpD, address:145.79.227.10:4425): protocol: read tcp 188.166.225.170:58348->145.79.227.10:4425: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598463220 (169h27m50.63803635s) [noise=true , address=145.79.227.35:4501 , piece_id=UKSF2KK675OI35QE6QZTH7QAJQ4SD7E42ZPMY2JIANOHGEFF44UQ, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:145.79.227.35:4501): protocol: rpc: dial tcp 145.79.227.35:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598484230 (169h27m50.63805735s) [noise=true , address=145.79.227.14:4867 , piece_id=7BW6OXJYXIXQSQK3DMHSGQ56ANOAOEGXEHD4EWCOZDXTTIL37SAA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12Sz7PyMo81CmAAkiCwpZ3M3JaM9HMHJLwAAM5M5kv6nnKktpKE, address:145.79.227.14:4867): protocol: read tcp 188.166.225.170:60252->145.79.227.14:4867: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598492155 (169h27m50.638065275s) [noise=true , address=145.79.227.18:4018 , piece_id=72FTA6TNIQAVS2NDA5XOR4J2ZEEBWZ7LLGT4EAQ2EN2YB3VXG5JA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:1bFRHbXjTaxoEKc5gbQWYt6jwUg8jknScBZpc8PdKNSZhYqUqP, address:145.79.227.18:4018): protocol: read tcp 188.166.225.170:60682->145.79.227.18:4018: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598496193 (169h27m50.638069313s) [noise=true , address=145.79.227.47:4286 , piece_id=D2PSYV2IN5MFZGGF2U47EPRTCNDIMHB74GR3UFQ64AJN37FE7BVA, seg_pos={0 0}, upload=148985]: Done uploading piece. err: ecclient: upload failed (node:12v2vJnE92VrJXQrbxnAxANHCVASuXiBKMUZECMVmEWEjsZRtsQ, address:145.79.227.47:4286): protocol: read tcp 188.166.225.170:33092->145.79.227.47:4286: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Sep 13 14:56:14.598521301 (169h27m50.638094421s) [seg_pos={0 0}, upload=148985]: Done waiting for segment. successful: 3 optimal: 14 errs: <nil>
Sep 13 14:56:14.862062924 (169h27m50.901636044s) [upload=148985]: Done uploading segments...
Sep 13 14:56:14.862090337 (169h27m50.901663457s) [upload=148985]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 3)
Duration: 17.565093611s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
8.974073472s
FAIL (2025-09-13 15:38:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.974148076s
core
uplink
us-select-1
us1-placement-20
|