Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 14.49796083s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:55:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.497979785s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....X.........X.....X.....X..............X.X................] duration: 3.557321093s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:54:56 UTC) orbiter warning: context deadline exceeded Duration: 3.557344928s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.....X.........X.....X.....X..............X.X...............] duration: 3.859366069s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:54:39 UTC) orbiter warning: context deadline exceeded Duration: 3.859419531s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 3.23010509s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:54:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.230126842s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................................X.X..X.............] duration: 6.481915571s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:54:15 UTC) orbiter warning: context deadline exceeded Duration: 6.481937121s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [......X.........X.....X.....X..............X.X..............] duration: 3.471291761s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:53:44 UTC) orbiter warning: context deadline exceeded Duration: 3.471318552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........................................X.X..X............] duration: 10.650334329s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:53:22 UTC) orbiter warning: context deadline exceeded Duration: 10.650441483s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.......X.........X.....X.....X..............X.X.............] duration: 4.160354409s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:53:21 UTC) orbiter warning: context deadline exceeded Duration: 4.160381491s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 2.768983419s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:53:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.769005921s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [........X.........X.....X.....X..............X.X............] duration: 5.000527525s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:53:04 UTC) orbiter warning: context deadline exceeded Duration: 5.000543395s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........................................X.X..X...........] duration: 9.333097453s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:52:45 UTC) orbiter warning: context deadline exceeded Duration: 9.333124773s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 3.01068014s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:52:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.010705448s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [.........X.........X.....X.....X..............X.X...........] duration: 3.535467432s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:52:30 UTC) orbiter warning: context deadline exceeded Duration: 3.535499132s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............................................X.X..X..........] duration: 5.940654701s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:52:13 UTC) orbiter warning: context deadline exceeded Duration: 5.940677706s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 2.396010478s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:52:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.396041747s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [..........X.........X.....X.....X..............X.X..........] duration: 3.474065s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:52:02 UTC) orbiter warning: context deadline exceeded Duration: 3.474197763s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............................................X.X..X.........] duration: 5.863790972s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:51:57 UTC) orbiter warning: context deadline exceeded Duration: 5.863822072s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [...........X.........X.....X.....X..............X.X.........] duration: 3.526038961s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:51:38 UTC) orbiter warning: context deadline exceeded Duration: 3.526063346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 3.295356199s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.295373001s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............................................X.X..X........] duration: 6.250210437s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:51:06 UTC) orbiter warning: context deadline exceeded Duration: 6.250237709s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 2.23922368s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:50:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.239252244s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [............X.........X.....X.....X..............X.X........] duration: 3.334976064s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:50:41 UTC) orbiter warning: context deadline exceeded Duration: 3.335005149s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 2.270350721s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:50:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.270382401s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............................................X.X..X.......] duration: 6.015216229s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:49:53 UTC) orbiter warning: context deadline exceeded Duration: 6.015240245s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [X............X.........X.....X.....X..............X.X.......] duration: 3.483373415s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:49:47 UTC) orbiter warning: context deadline exceeded Duration: 3.483397652s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [.X............X.........X.....X.....X..............X.X......] duration: 3.386710094s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:49:24 UTC) orbiter warning: context deadline exceeded Duration: 3.386732767s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.185159001s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:49:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.185260644s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................................................X.X..X......] duration: 7.720349646s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:48:49 UTC) orbiter warning: context deadline exceeded Duration: 7.720381297s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.6942759s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:48:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.694358148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.697139312s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:48:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.697167476s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 2.292557766s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:48:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.292583095s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [X.X............X.........X.....X.....X..............X.X.....] duration: 3.625481002s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:48:15 UTC) orbiter warning: context deadline exceeded Duration: 3.626187719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.223649986s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:47:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.223700983s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.841938752s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:47:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.841967116s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.X.X............X.........X.....X.....X..............X.X....] duration: 5.00044526s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:47:54 UTC) orbiter warning: context deadline exceeded Duration: 5.000468314s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................................................X.X..X.....] duration: 6.015898441s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:47:37 UTC) orbiter warning: context deadline exceeded Duration: 6.015922305s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..X.X............X.........X.....X.....X..............X.X...] duration: 3.442539311s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:47:36 UTC) orbiter warning: context deadline exceeded Duration: 3.442568176s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 2.22700268s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:47:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.227029751s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.700168266s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:47:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.700194487s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...X.X............X.........X.....X.....X..............X.X..] duration: 3.60523581s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:46:43 UTC) orbiter warning: context deadline exceeded Duration: 3.605259046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 2.283044725s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:46:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.28307399s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................................................X.X..X....] duration: 6.005500964s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:46:29 UTC) orbiter warning: context deadline exceeded Duration: 6.005537153s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [....X.X............X.........X.....X.....X..............X.X.] duration: 3.419012523s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:46:11 UTC) orbiter warning: context deadline exceeded Duration: 3.419040707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.764442316s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:46:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.764472404s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.267752293s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:45:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.267780398s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................................................X.X..X...] duration: 6.00664462s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:45:34 UTC) orbiter warning: context deadline exceeded Duration: 6.006668174s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.672046582s trace: https://jaeger.dev.storj.io/trace/722611fbdbaed78e FAIL (2025-12-25 16:45:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q) context deadline exceeded; piecestore: (Node ID: 1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, Piece ID: BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ) context deadline exceeded; piecestore: (Node ID: 12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, Piece ID: VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA) context deadline exceeded; piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA) context deadline exceeded ======LOGS====== 6:04:39.581206993 (480h35m39.452290028s) [upload=421859]: Starting upload Dec 25 16:04:39.581286003 (480h35m39.452369048s) [upload=421859]: Done starting upload Dec 25 16:04:39.581530759 (480h35m39.452613794s) [upload=421859]: Uploading segments... Dec 25 16:04:39.581610911 (480h35m39.452693946s) [upload=421859]: Got next segment. Inline: false Dec 25 16:04:39.581640698 (480h35m39.452723743s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 16:04:39.581645397 (480h35m39.452728442s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 16:04:39.826079250 (480h35m39.697162285s) [seg_pos={0 0}, upload=421859]: Begin upload segment... Dec 25 16:04:39.826129897 (480h35m39.697212932s) [seg_pos={0 0}, upload=421859]: Done begin upload segment. Dec 25 16:04:39.826171075 (480h35m39.697254111s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826182347 (480h35m39.697265382s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826232663 (480h35m39.697315708s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.826651290 (480h35m39.697734335s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827034740 (480h35m39.698117785s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827420354 (480h35m39.69850341s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.827819594 (480h35m39.698902639s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828276645 (480h35m39.69935969s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.828810451 (480h35m39.699893487s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830214384 (480h35m39.70129742s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.830323522 (480h35m39.701406567s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831497668 (480h35m39.702580703s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.831727687 (480h35m39.702810732s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.832764601 (480h35m39.703847636s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.834102589 (480h35m39.705185634s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:39.835396242 (480h35m39.706479287s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Uploading piece... Dec 25 16:04:41.619760569 (480h35m41.490843604s) [upload=421859]: Next returned nil segment Dec 25 16:04:41.619775707 (480h35m41.490858752s) [upload=421859]: Waiting for error group managing segments... Dec 25 16:04:41.989065086 (480h35m41.860148131s) [noise=true , address=145.79.227.50:4374 , piece_id=BJW5TQSENHNKOLFPKXMFWO5D5OKI6VMQODYFUC2WUIEN3JE4RSOA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:41.999186809 (480h35m41.870269844s) [noise=true , address=145.79.227.5:4222 , piece_id=HRU2IYFK7USA3PU3RUIFRLFVLJXXVT76EEJDUDRIJIOR4KDL5ZTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.002386692 (480h35m41.873469737s) [noise=true , address=145.79.227.45:4231 , piece_id=TKUTBXO3IVYXQBLIBHTTLEBD5VPQVHM36OUFQEMSCKVWQZ2DNISA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.003731243 (480h35m41.874814278s) [noise=true , address=145.79.227.42:4122 , piece_id=X2NUXAG46PUZEH2U3MXSOFLSHBGQRL4HWQOAJ3NZR2ZLVBDEHIQA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.005531150 (480h35m41.876614195s) [noise=true , address=45.91.116.6:4973 , piece_id=WP4KJU2IGJHZRXTPI3IF23TR3S6B26OIKMEWXGVY2OW57RAEENBA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.014159270 (480h35m41.885242315s) [noise=true , address=145.79.227.13:4804 , piece_id=WBGXIWOFCP3NSQHB3AACUODXSLSFMTH5XDVIGXGAE254EUETHHKA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.018698233 (480h35m41.889781278s) [noise=true , address=145.79.227.11:4729 , piece_id=ILQZ6HZOTW3CC5WLRJHRP64LGRHHGMAUYIRISIZORYFLDI3464FQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.036598427 (480h35m41.907681492s) [noise=true , address=145.79.227.41:4086 , piece_id=AHGFZ47MPT2VTAG52EYBPKMGYM3IJVNBLKR6WYBXD5QPNVLWVZEQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.070316953 (480h35m41.941399988s) [noise=true , address=145.79.227.4:4186 , piece_id=LY5W7DRT74JDRUIFVMQBCEN5EKZD6ODPIHQN34E72HVZL7OTLIYQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.307154572 (480h35m42.178237617s) [noise=true , address=45.91.116.12:4227 , piece_id=VKUB764KKKRBZZUY6IEIPNF7AKUREYZHMY76QMBKAPWHXKANAPAA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.310278521 (480h35m42.181361556s) [noise=true , address=145.79.227.23:4545 , piece_id=VUT5GABA4VUIXPSG2LFTGMRVOQ54RCBUPP2Q565DM5DDVXR6EQYA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.409967535 (480h35m42.28105058s) [noise=true , address=145.79.227.2:4114 , piece_id=BJIKN5I46YXBARQYFN7SO3F2VL45XNXJQ3CF2SPRDTCMZUHS5VTQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424848550 (480h35m42.295931596s) [noise=true , address=45.91.116.9:4261 , piece_id=SHX7UJRA2UXQV6L5MRD3HHPL56BXXE27SATK27BH6HCSVFHWT27Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: <nil> Dec 25 16:04:42.424871044 (480h35m42.295954089s) [seg_pos={0 0}, upload=421859]: Segment reached optimal threshold of 13 pieces. Dec 25 16:04:42.425106202 (480h35m42.296189247s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125GxLsL5UtXkuJ1LFvF42N9iZkKtRK3rq63xdR25JQbtwoT1L5): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425224757 (480h35m42.296307832s) [noise=true , address=145.79.227.46:4601 , piece_id=4USZDSIL7ITQLPX7S7B2H3EWC2UUR3HRWWXI52GEJLDDVOGNS7FA, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425246729 (480h35m42.296329774s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 25 16:04:42.425255015 (480h35m42.29633806s) [noise=true , address=45.91.116.13:4156 , piece_id=BKONGNPQ7HHJULHBCKTLWPOUEKB5AAAXZ75OYCXS7JXMZ7I4D3NQ, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425525610 (480h35m42.296608645s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36512->145.79.227.49:4338: use of closed network connection Dec 25 16:04:42.425535539 (480h35m42.296618574s) [noise=true , address=145.79.227.49:4338 , piece_id=AT5PBPZ5E6ZKV2YXNTXNEFXEN3IM6WGSOB64X54IAX54VV2OO54Q, seg_pos={0 0}, upload=421859]: Long tail cancellation due to optimal threshold Dec 25 16:04:42.425555858 (480h35m42.296638903s) [seg_pos={0 0}, upload=421859]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 16:04:42.425640328 (480h35m42.296723373s) [upload=421859]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 16:04:42.425647221 (480h35m42.296730266s) [upload=421859]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 16:04:42.716492268 (480h35m42.587575304s) [upload=421859]: Done uploading segments... Dec 25 16:04:42.716508891 (480h35m42.587591926s) [upload=421859]: Upload finished. err: <nil> Duration: 5.672076308s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.....X.X............X.........X.....X.....X..............X.X] duration: 5.00777066s trace: https://jaeger.dev.storj.io/trace/674fb97c28501bcd FAIL (2025-12-25 16:45:15 UTC) orbiter warning: context deadline exceeded Duration: 5.007799794s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................................................X.X..X..] duration: 6.547603875s trace: https://jaeger.dev.storj.io/trace/8d6c9ad46c081fb FAIL (2025-12-25 16:44:48 UTC) orbiter warning: context deadline exceeded Duration: 6.547631316s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.230223461s trace: https://jaeger.dev.storj.io/trace/5b16c8817ce751d7 FAIL (2025-12-25 16:44:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.230246995s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)