Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
5.962942582s
FAIL (2025-07-02 23:03:01 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.962963573s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
5.980532837s
FAIL (2025-07-02 23:02:44 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.980553886s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
6.255188099s
FAIL (2025-07-02 23:01:55 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.255208878s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.356735523s
FAIL (2025-07-02 23:01:37 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.356759378s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.238885281s
FAIL (2025-07-02 23:00:32 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.238901622s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.078088434s
FAIL (2025-07-02 22:59:22 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.07811234s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.085798532s
FAIL (2025-07-02 22:58:50 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.085821846s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.22194934s
FAIL (2025-07-02 22:58:28 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.222003392s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
5.945000186s
FAIL (2025-07-02 22:58:09 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.945027227s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
6.234996417s
FAIL (2025-07-02 22:57:46 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.235024832s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.041905533s
FAIL (2025-07-02 22:57:01 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.041933026s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
5.544451465s
FAIL (2025-07-02 22:56:42 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.544510008s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.116174736s
FAIL (2025-07-02 22:55:57 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.116197229s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
5.93298985s
FAIL (2025-07-02 22:55:07 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.933011822s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.189970778s
FAIL (2025-07-02 22:54:47 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.189989554s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.209477089s
FAIL (2025-07-02 22:53:57 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.209556159s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.062153749s
FAIL (2025-07-02 22:53:20 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.062208914s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
5.989186645s
FAIL (2025-07-02 22:52:17 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.989209558s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
6.210594304s
FAIL (2025-07-02 22:51:10 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.210615966s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
5.897565898s
FAIL (2025-07-02 22:49:56 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.897595194s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
6.061446404s
FAIL (2025-07-02 22:49:13 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.061473846s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
6.278626469s
FAIL (2025-07-02 22:48:57 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.278833354s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
5.977910382s
FAIL (2025-07-02 22:47:50 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.977931132s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.095708915s
FAIL (2025-07-02 22:46:43 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.095725365s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.393974075s
FAIL (2025-07-02 22:45:34 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.394037146s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
5.796572487s
FAIL (2025-07-02 22:44:44 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.796595964s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.018107921s
FAIL (2025-07-02 22:43:50 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.01813295s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.059672143s
FAIL (2025-07-02 22:43:28 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.059698324s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
5.885513466s
FAIL (2025-07-02 22:42:42 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.885534957s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.266491208s
FAIL (2025-07-02 22:41:30 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.266511117s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.291829961s
FAIL (2025-07-02 22:41:12 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.291859879s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.181970671s
FAIL (2025-07-02 22:40:08 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.181995829s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.038551826s
FAIL (2025-07-02 22:39:21 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.038887267s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.156635778s
FAIL (2025-07-02 22:38:34 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.156653653s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.188435923s
FAIL (2025-07-02 22:37:21 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.188458275s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.249263233s
FAIL (2025-07-02 22:36:17 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.249280906s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.956392147s
FAIL (2025-07-02 22:35:51 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 5.956407488s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.15572551s
FAIL (2025-07-02 22:34:50 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.155747542s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
6.890061362s
FAIL (2025-07-02 22:34:06 UTC)
orbiter warning: read tcp 188.166.225.170:47910->45.38.24.2:443: read: connection reset by peer
Duration: 6.890082031s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.622830182s
FAIL (2025-07-02 19:41:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.622857345s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.699257391s
FAIL (2025-07-02 19:40:39 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.699281115s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.550028152s
FAIL (2025-07-02 19:39:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.550071685s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.652385986s
FAIL (2025-07-02 19:39:25 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.652426554s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.6810146s
FAIL (2025-07-02 19:39:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.681042642s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
7.80240631s
FAIL (2025-07-02 19:38:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 7.802428071s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.902852994s
FAIL (2025-07-02 19:37:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.902879944s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.690597617s
FAIL (2025-07-02 19:36:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.690648995s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.69898311s
FAIL (2025-07-02 19:36:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.699008319s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
5.745853611s
FAIL (2025-07-02 19:35:08 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.745881353s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.655040529s
FAIL (2025-07-02 19:34:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
tItem_SegmentCommit
Jul 2 18:58:54.013093667 (225h48m28.783735003s) [upload=180606]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jul 2 18:58:54.309561686 (225h48m29.080203011s) [upload=180606]: Done uploading segments...
Jul 2 18:58:54.309581624 (225h48m29.080222959s) [upload=180606]: Upload finished. err: <nil>
Jul 2 19:00:04.201216278 (225h49m38.971857623s) [upload=180623]: Starting upload
Jul 2 19:00:04.201266945 (225h49m38.97190828s) [upload=180623]: Done starting upload
Jul 2 19:00:04.201441849 (225h49m38.972083194s) [upload=180623]: Uploading segments...
Jul 2 19:00:04.201470774 (225h49m38.972112119s) [upload=180623]: Got next segment. Inline: false
Jul 2 19:00:04.201497265 (225h49m38.97213861s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 2 19:00:04.201502154 (225h49m38.972143489s) [upload=180623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 2 19:00:04.435725278 (225h49m39.206366623s) [seg_pos={0 0}, upload=180623]: Begin upload segment...
Jul 2 19:00:04.435786185 (225h49m39.20642752s) [seg_pos={0 0}, upload=180623]: Done begin upload segment.
Jul 2 19:00:04.435808808 (225h49m39.206450153s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435819870 (225h49m39.206461215s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.435888140 (225h49m39.206529485s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436417083 (225h49m39.207058428s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.436873325 (225h49m39.20751467s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437214307 (225h49m39.207855642s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437538487 (225h49m39.208179832s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.437909017 (225h49m39.208550362s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.438770745 (225h49m39.20941208s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439143538 (225h49m39.209784873s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439509218 (225h49m39.210150563s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.439865549 (225h49m39.210506884s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440200541 (225h49m39.210841876s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440568144 (225h49m39.211209479s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440922372 (225h49m39.211563717s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:04.440994419 (225h49m39.211635754s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Uploading piece...
Jul 2 19:00:06.097759361 (225h49m40.868400696s) [upload=180623]: Next returned nil segment
Jul 2 19:00:06.097783598 (225h49m40.868424943s) [upload=180623]: Waiting for error group managing segments...
Jul 2 19:00:06.362437140 (225h49m41.133078476s) [noise=true , address=104.36.82.15:4259 , piece_id=YBK656SAIK4VZ6FYQOZSEELXJ442TM33UMUUDG6AGAP3ASUURXWQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.386050957 (225h49m41.156692292s) [noise=true , address=104.36.82.6:4875 , piece_id=PQ56TJVVVRS53FZOE3WMHJOVXQBFY5TWJZZM7CZVUCPQZQOHHVGQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.400244451 (225h49m41.170885796s) [noise=true , address=104.36.82.8:4970 , piece_id=GNQQF7YMOLDBZQRFQVATLH6JPYO6MDZPLJYYRTIHYUMEIPADJORQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.411574206 (225h49m41.182215551s) [noise=true , address=104.36.82.16:4295 , piece_id=65QLBQIQP5VY6PFHTAICM2F2REPP6U27PFOXDTJGEX7YMYW2WKUQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.423389331 (225h49m41.194030666s) [noise=true , address=104.36.82.12:4149 , piece_id=IAYCDUF7BXE3SBFZKWDY4BYKX4FJND7OS65QG4BM5I63YGT63I4Q, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.466237456 (225h49m41.236878801s) [noise=true , address=104.36.82.13:4186 , piece_id=WZ2UYYZKGWWCD6R5SWT6V4A3QXT7ZFPR27GRYUZW3ECQMHHOQFDA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.473676578 (225h49m41.244317923s) [noise=true , address=104.36.82.4:4801 , piece_id=CDSGC6HKVBJPHYZOFGH2BPXNFAIQFP3ROWRY27YZLZDRINAD72WQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:06.548500307 (225h49m41.319141652s) [noise=true , address=104.36.82.8:4971 , piece_id=6UVTLNKSPYWUFXUYNR54ALHJWF3MRX3JYTOZZOLMSWIMKDMEYX2A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:15.665518789 (225h49m50.436160124s) [noise=true , address=45.91.116.14:4190 , piece_id=FR4WUKCNTHGI7MFL2VANU3PNYZHXCDBEE4YCQYAMZJRBRUFPC6YA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:21.814339372 (225h49m56.584980697s) [noise=true , address=45.91.116.12:4226 , piece_id=4U2FRI7N5P6OLJ77TWP5NWVXSRKYLEGCFPBD3HXS5IM6WZREDIIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:22.509711564 (225h49m57.280352899s) [noise=true , address=45.91.116.4:4937 , piece_id=J4IRPW5WRLWNJYNYRT2S56FTVYCVUXUPXD2IBRRLYH4MAGHYZVZQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: <nil>
Jul 2 19:00:33.550625303 (225h50m8.321266638s) [noise=true , address=45.91.116.11:4118 , piece_id=AHHUQZNNRODQERLXMVHCQ4MMVRK2D2YCQSYPQ4GCWCFGF66VSC6A, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, address:45.91.116.11:4118): protocol: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55876->45.91.116.11:4118: use of closed network connection
Jul 2 19:00:33.550781382 (225h50m8.321422707s) [noise=true , address=77.90.31.13:4627 , piece_id=C6UIMNH4WMIPIF2DQXLARIWCDUWM7NJHHONCHZT7YXVXT6OSOCIQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: read tcp 188.166.225.170:33268->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550813153 (225h50m8.321454488s) [noise=true , address=77.90.31.18:4805 , piece_id=PFLMSIA4H6HLIBVJMSJ2WUZAPSM2OCKVQT7SIRAOYZ435LQIMXNA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 188.166.225.170:53132->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550844112 (225h50m8.321485447s) [noise=true , address=77.90.31.21:4879 , piece_id=3MX4ZCSAUGOJOGKMA624CTJUTZWUQCTVDFGNWLU32FLSFTELA7BQ, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:12W5Bso6XAuE6qzduTxCUmuzHWnUvNrjiwvogysvBLisdHEFjQQ, address:77.90.31.21:4879): protocol: read tcp 188.166.225.170:50994->77.90.31.21:4879: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550874289 (225h50m8.321515614s) [noise=true , address=77.90.31.14:4663 , piece_id=LERBTUH5FX2R33DOFNHDWTYHMCXPMUFVWNZACT4QV5N2LOED54TA, seg_pos={0 0}, upload=180623]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: read tcp 188.166.225.170:36232->77.90.31.14:4663: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 2 19:00:33.550886583 (225h50m8.321527918s) [seg_pos={0 0}, upload=180623]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Jul 2 19:00:33.791812641 (225h50m8.562453976s) [upload=180623]: Done uploading segments...
Jul 2 19:00:33.791848710 (225h50m8.562490045s) [upload=180623]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 5.655210765s
core
uplink
us-select-1
|