Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.X....................................] duration: 2.306332486s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:19:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.306364647s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................X.X...................................] duration: 2.358306948s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:19:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.358335293s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.X..................................] duration: 2.346283561s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:18:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.346316865s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................X.X.................................] duration: 2.37062868s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:18:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.370710134s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................X.X................................] duration: 2.298537863s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:17:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.298620451s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.X...............................] duration: 2.598775103s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:16:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.598803638s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................X.X..............................] duration: 2.319692286s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:16:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.319747821s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................X.X.............................] duration: 2.408866983s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:15:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.408901078s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................X.X............................] duration: 2.35412314s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:15:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.354204784s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................X.X...........................] duration: 2.393329047s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:14:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.393357831s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................X.X..........................] duration: 2.314578802s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:14:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.31466697s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................X.X.........................] duration: 2.328295009s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:12:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.328389449s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................X.X........................] duration: 2.293262238s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:12:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.293299518s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................X.X.......................] duration: 2.314129086s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:10:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.314159874s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................X.X......................] duration: 2.325761202s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:10:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.325795246s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................X.X.....................] duration: 2.3217269s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:09:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.321760375s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................X.X....................] duration: 2.319134083s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:09:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.319159781s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................X.X...................] duration: 2.378134943s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:08:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.378165752s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................X.X..................] duration: 2.370636015s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:07:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.370664309s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................X.X.................] duration: 2.390180834s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:07:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.390212244s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X.X................] duration: 2.363126927s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:06:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.363175641s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X.X...............] duration: 2.27293037s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:06:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.272955739s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........................................X.X..............] duration: 2.355392268s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:05:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.355418157s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [............................................X.X.............] duration: 2.258597908s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:05:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.258629989s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............................................X.X............] duration: 2.353627586s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:04:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.35365631s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............................................X.X...........] duration: 2.289194922s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:03:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.289231991s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............................................X.X..........] duration: 2.31989876s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:02:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.319933047s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [................................................X.X.........] duration: 2.478506286s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:01:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.478536955s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................................................X.X........] duration: 2.291266297s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:01:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.291329828s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..................................................X.X.......] duration: 2.339222207s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 09:01:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.339255019s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [...................................................X.X......] duration: 2.333311919s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:59:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.333363297s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [....................................................X.X.....] duration: 2.444454182s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:59:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.444589229s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....................................................X.X....] duration: 2.364370906s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:58:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.36439513s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.791574546s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:57:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.791603131s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X.X...] duration: 2.717467272s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:57:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.717499163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.748513121s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:56:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.748545351s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X.X..] duration: 2.37857366s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:55:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.378606051s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.895022784s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:55:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.895069343s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X.X.] duration: 2.436131672s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:54:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.436146591s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.843775907s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:54:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.843810462s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................X.X] duration: 30.001860762s trace: https://jaeger.dev.storj.io/trace/47bdbcdc6f22899b FAIL (2025-11-11 08:54:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001889016s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.805620353s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:53:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.805649459s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 2.413581223s trace: https://jaeger.dev.storj.io/trace/1456b5d1381f3a2f FAIL (2025-11-11 08:53:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.413655955s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.886928955s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:53:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.886957158s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002092942s trace: https://jaeger.dev.storj.io/trace/1456b5d1381f3a2f FAIL (2025-11-11 08:53:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.002121777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.944205256s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.944224172s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.030248621s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:51:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.030278839s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.682560543s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:50:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.682589679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.879307514s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:49:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 5.879349904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.071650011s trace: https://jaeger.dev.storj.io/trace/49621e5e6beafe8e FAIL (2025-11-11 08:48:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Nov 11 08:10:09.433180409 (48h6m7.98465087s) [upload=42408]: Starting upload Nov 11 08:10:09.433429884 (48h6m7.984900335s) [upload=42408]: Done starting upload Nov 11 08:10:09.433567967 (48h6m7.985038438s) [upload=42408]: Uploading segments... Nov 11 08:10:09.434191213 (48h6m7.985661674s) [upload=42408]: Got next segment. Inline: false Nov 11 08:10:09.434274032 (48h6m7.985744493s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 11 08:10:09.434282217 (48h6m7.985752688s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 11 08:10:09.714872160 (48h6m8.266342621s) [seg_pos={0 0}, upload=42408]: Begin upload segment... Nov 11 08:10:09.715084585 (48h6m8.266555046s) [seg_pos={0 0}, upload=42408]: Done begin upload segment. Nov 11 08:10:09.715464638 (48h6m8.2669351s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.715555221 (48h6m8.267025692s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718085410 (48h6m8.269555901s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.718721901 (48h6m8.270192402s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.719505724 (48h6m8.270976195s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.720136295 (48h6m8.271606756s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721166056 (48h6m8.272636517s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.721883453 (48h6m8.273353994s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.722398593 (48h6m8.273869054s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.724101306 (48h6m8.275571767s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.726385264 (48h6m8.277855776s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.728323225 (48h6m8.279793686s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.730321541 (48h6m8.281792002s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.731324300 (48h6m8.282794832s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.732226989 (48h6m8.28369749s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:09.733026101 (48h6m8.284496562s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Uploading piece... Nov 11 08:10:11.398415378 (48h6m9.949885839s) [upload=42408]: Next returned nil segment Nov 11 08:10:11.398434043 (48h6m9.949904504s) [upload=42408]: Waiting for error group managing segments... Nov 11 08:10:11.796154582 (48h6m10.347625043s) [noise=true , address=77.90.31.18:4806 , piece_id=LNL5LRHH3OOELKEO4HND2K3GNVDPCBCGKBNJ75DCLEKRAOMTQLKA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.809043214 (48h6m10.360513675s) [noise=true , address=45.91.116.8:4047 , piece_id=VP6VW2ZTNSKS35R47GGJZI4YSHNNFUFB4YQ3W3QFT5ENN7F73VQA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.866335992 (48h6m10.417806453s) [noise=true , address=45.91.116.11:4117 , piece_id=UOEFPOPOFIO5KXY2JOX54YHUWNMTCAFMXEBKAMMR4GSR5ZE6J6PA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:11.932922087 (48h6m10.484392538s) [noise=true , address=77.90.31.19:4841 , piece_id=6JVB2WIEA2STMNBVTM6D7IYFHQY4QJGXAYHJAQILMYZZDH2MWIVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:12.005905333 (48h6m10.557375794s) [noise=true , address=45.91.116.4:4939 , piece_id=KMHEQSQTA2WEWMIO7JDQZUDLSRV4CA6OJUN2CYYUSA7SU2ZDQBCQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:16.703757864 (48h6m15.255228326s) [noise=true , address=145.79.227.9:4366 , piece_id=JDBR4PTBZ5RN6NTWPERVLFANUAZQF3FSGIENVRHSYN56ED5NT2AQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:17.540298352 (48h6m16.091768813s) [noise=true , address=145.79.227.21:4474 , piece_id=RNB5T6DIUCY7IV2ULLO2UJRDKUJXI34UM4ZCHPJB22DNQ47VZJVA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:19.805526866 (48h6m18.356997337s) [noise=true , address=145.79.227.27:4689 , piece_id=KLTJYVDWHEE2BGLUSP2XT6L5R2C5OB7TF4DZLYBF6R5OG3N6AEFQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.397542313 (48h6m22.949012764s) [noise=true , address=145.79.227.17:4970 , piece_id=F2227LWTX5PRCQIUEJNRJ3QS3GRJOIEJIZFV3KRF7BAEAPSQZR7A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:24.779686050 (48h6m23.331156511s) [noise=true , address=145.79.227.14:4840 , piece_id=QASWXZB3E6E7VYGF24KZ37Q3LKT6F4WVFKTBV3YS73KYAWTBV7YQ, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:25.298168925 (48h6m23.849639387s) [noise=true , address=145.79.227.11:4729 , piece_id=CDIU7FFPCVQGDK3K5ZDUARJ6VGB7KIY2DEVL7ZRLW5EASIFEHKHA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:35.044925402 (48h6m33.596395853s) [noise=true , address=145.79.227.30:4761 , piece_id=ASO5O5LM6CEF6DXGBVOCI3NOCWLGKDU2T5JHOL5ZPWEXMTRMKZIA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306959937 (48h6m35.858430398s) [noise=true , address=145.79.227.20:4437 , piece_id=AOK6MTRZKPCXTNHZWGBYJGRBYNHFWO7GSG2PT4DFSMDAWK75GY4A, seg_pos={0 0}, upload=42408]: Done uploading piece. err: <nil> Nov 11 08:10:37.306987550 (48h6m35.858458011s) [seg_pos={0 0}, upload=42408]: Segment reached optimal threshold of 13 pieces. Nov 11 08:10:37.307209231 (48h6m35.858679702s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ACUxMRYJG7bzfXw6i1ioF6FSzbQvuWmZgrgcH8XhCxcncycL4): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307218759 (48h6m35.85868922s) [noise=true , address=145.79.227.28:4726 , piece_id=ADS5YQCWYXWSVRAOSHIXR3CDLMWOELEBE3TUVDB4KML4VXESCWPA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307368936 (48h6m35.858839397s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw): protocol: expected piece hash; context canceled; EOF Nov 11 08:10:37.307376741 (48h6m35.858847202s) [noise=true , address=145.79.227.25:4617 , piece_id=4RZT3DJ4PSOAKSBGNLV4BZE2MMWEB5677474SIVSSEK6EIQZCXSA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307561282 (48h6m35.859031743s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12touQ91P4kPaTdodCZ2EfbbNE84Yto2kYSPhQ6UMZTR5mtH5r9): protocol: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46680->145.79.227.22:4510: use of closed network connection Nov 11 08:10:37.307585969 (48h6m35.85905643s) [noise=true , address=145.79.227.22:4510 , piece_id=NQNGJKB5MZMR7U5DXVBEA6LGGJCVTLHH2HEIVJKI2CS54C4NLQGA, seg_pos={0 0}, upload=42408]: Long tail cancellation due to optimal threshold Nov 11 08:10:37.307629221 (48h6m35.859099682s) [seg_pos={0 0}, upload=42408]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Nov 11 08:10:37.307741005 (48h6m35.859211466s) [upload=42408]: Flush batch item: pb.BatchRequestItem_SegmentCommit Nov 11 08:10:37.307750052 (48h6m35.859220513s) [upload=42408]: Flush batch item: pb.BatchRequestItem_ObjectCommit Nov 11 08:10:37.612339386 (48h6m36.163809848s) [upload=42408]: Done uploading segments... Nov 11 08:10:37.612366317 (48h6m36.163836788s) [upload=42408]: Upload finished. err: <nil> Duration: 6.071676391s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)