Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......XX.X....X....X.XX...XXXXXX....X...XX...X.X......X.XX..] duration: 4.838843735s trace: https://jaeger.dev.storj.io/trace/31d256189513d4a3 FAIL (2025-12-25 13:03:31 UTC) orbiter warning: context deadline exceeded Duration: 4.838891866s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......XX.X....X....X.XX...XXXXXX....X...XX...X.X......X.XX.] duration: 3.555950964s trace: https://jaeger.dev.storj.io/trace/31d256189513d4a3 FAIL (2025-12-25 13:02:18 UTC) orbiter warning: context deadline exceeded Duration: 3.555971693s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [........XX.X....X....X.XX...XXXXXX....X...XX...X.X......X.XX] duration: 5.001287462s trace: https://jaeger.dev.storj.io/trace/31d256189513d4a3 FAIL (2025-12-25 13:01:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001309165s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X........XX.X....X....X.XX...XXXXXX....X...XX...X.X......X.X] duration: 5.002377077s trace: https://jaeger.dev.storj.io/trace/1027afb6c49a3f46 FAIL (2025-12-25 13:00:28 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00240461s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [.X........XX.X....X....X.XX...XXXXXX....X...XX...X.X......X.] duration: 4.558496443s trace: https://jaeger.dev.storj.io/trace/16796d4f3efb0e0c FAIL (2025-12-25 12:59:36 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.55852031s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 20 out of 60 is failed (threshold=6) [..X........XX.X....X....X.XX...XXXXXX....X...XX...X.X......X] duration: 5.001793758s trace: https://jaeger.dev.storj.io/trace/16796d4f3efb0e0c FAIL (2025-12-25 12:59:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001816942s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [...X........XX.X....X....X.XX...XXXXXX....X...XX...X.X......] duration: 4.219973973s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:58:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.220040681s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [....X........XX.X....X....X.XX...XXXXXX....X...XX...X.X.....] duration: 5.001233389s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:58:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001259008s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.....X........XX.X....X....X.XX...XXXXXX....X...XX...X.X....] duration: 3.466387717s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:57:50 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.466415881s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [......X........XX.X....X....X.XX...XXXXXX....X...XX...X.X...] duration: 3.447213597s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:57:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.447231721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.......X........XX.X....X....X.XX...XXXXXX....X...XX...X.X..] duration: 4.444165729s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:57:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.444192871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [........X........XX.X....X....X.XX...XXXXXX....X...XX...X.X.] duration: 3.883566026s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:56:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.883598337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 19 out of 60 is failed (threshold=6) [.........X........XX.X....X....X.XX...XXXXXX....X...XX...X.X] duration: 5.001250714s trace: https://jaeger.dev.storj.io/trace/5f5343c5c809f8c1 FAIL (2025-12-25 12:55:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001273178s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [..........X........XX.X....X....X.XX...XXXXXX....X...XX...X.] duration: 4.933054268s trace: https://jaeger.dev.storj.io/trace/654960f5d3df1826 FAIL (2025-12-25 12:55:40 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.933078494s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 18 out of 60 is failed (threshold=6) [...........X........XX.X....X....X.XX...XXXXXX....X...XX...X] duration: 5.001080068s trace: https://jaeger.dev.storj.io/trace/654960f5d3df1826 FAIL (2025-12-25 12:54:30 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001105596s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [............X........XX.X....X....X.XX...XXXXXX....X...XX...] duration: 4.22064896s trace: https://jaeger.dev.storj.io/trace/6fabd37c37a39b26 FAIL (2025-12-25 12:53:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.220671893s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [.............X........XX.X....X....X.XX...XXXXXX....X...XX..] duration: 3.506887325s trace: https://jaeger.dev.storj.io/trace/6fabd37c37a39b26 FAIL (2025-12-25 12:52:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.506944805s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [..............X........XX.X....X....X.XX...XXXXXX....X...XX.] duration: 3.867453322s trace: https://jaeger.dev.storj.io/trace/6fabd37c37a39b26 FAIL (2025-12-25 12:52:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.86747309s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 17 out of 60 is failed (threshold=6) [...............X........XX.X....X....X.XX...XXXXXX....X...XX] duration: 5.001696588s trace: https://jaeger.dev.storj.io/trace/6fabd37c37a39b26 FAIL (2025-12-25 12:52:04 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001718509s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 16 out of 60 is failed (threshold=6) [................X........XX.X....X....X.XX...XXXXXX....X...X] duration: 5.001319445s trace: https://jaeger.dev.storj.io/trace/71b42df32c557543 FAIL (2025-12-25 12:51:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001336658s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [.................X........XX.X....X....X.XX...XXXXXX....X...] duration: 3.790787738s trace: https://jaeger.dev.storj.io/trace/5bcfd42d0f1ca881 FAIL (2025-12-25 12:50:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.790812324s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [..................X........XX.X....X....X.XX...XXXXXX....X..] duration: 4.494858068s trace: https://jaeger.dev.storj.io/trace/5bcfd42d0f1ca881 FAIL (2025-12-25 12:49:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.494882685s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [...................X........XX.X....X....X.XX...XXXXXX....X.] duration: 3.49463764s trace: https://jaeger.dev.storj.io/trace/5bcfd42d0f1ca881 FAIL (2025-12-25 12:48:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49466375s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 15 out of 60 is failed (threshold=6) [....................X........XX.X....X....X.XX...XXXXXX....X] duration: 5.000915356s trace: https://jaeger.dev.storj.io/trace/5bcfd42d0f1ca881 FAIL (2025-12-25 12:48:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000936947s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.....................X........XX.X....X....X.XX...XXXXXX....] duration: 5.0010062s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:47:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001029994s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [......................X........XX.X....X....X.XX...XXXXXX...] duration: 3.675538136s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:46:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.675590456s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.......................X........XX.X....X....X.XX...XXXXXX..] duration: 3.515184445s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:45:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.515270288s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [........................X........XX.X....X....X.XX...XXXXXX.] duration: 4.200335217s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:44:24 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.200362538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 14 out of 60 is failed (threshold=6) [.........................X........XX.X....X....X.XX...XXXXXX] duration: 5.000990439s trace: https://jaeger.dev.storj.io/trace/7f89a5f0e1e5e9d9 FAIL (2025-12-25 12:43:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001009284s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [..........................X........XX.X....X....X.XX...XXXXX] duration: 5.001471364s trace: https://jaeger.dev.storj.io/trace/10bf9b527e233fbc FAIL (2025-12-25 12:43:14 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001538922s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...........................X........XX.X....X....X.XX...XXXX] duration: 5.002040629s trace: https://jaeger.dev.storj.io/trace/785904e571cce3f6 FAIL (2025-12-25 12:42:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.002061569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [............................X........XX.X....X....X.XX...XXX] duration: 5.001070843s trace: https://jaeger.dev.storj.io/trace/4d1ad37aa26b6126 FAIL (2025-12-25 12:41:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00109549s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.4450596s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:40:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.445084888s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.............................X........XX.X....X....X.XX...XX] duration: 5.001059319s trace: https://jaeger.dev.storj.io/trace/813426f4ba5dee3 FAIL (2025-12-25 12:40:41 UTC) orbiter warning: context deadline exceeded Duration: 5.001079497s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.503683661s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:40:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.50371422s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..............................X........XX.X....X....X.XX...X] duration: 5.001275293s trace: https://jaeger.dev.storj.io/trace/64c8075328cd1a4f FAIL (2025-12-25 12:40:12 UTC) orbiter warning: context deadline exceeded Duration: 5.001300431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.336426977s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.336454159s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.856144364s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:39:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.856174502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...............................X........XX.X....X....X.XX...] duration: 5.000771576s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:39:12 UTC) orbiter warning: context deadline exceeded Duration: 5.000795551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [................................X........XX.X....X....X.XX..] duration: 3.562732706s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:38:22 UTC) orbiter warning: context deadline exceeded Duration: 3.562754157s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.549456841s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:38:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.54951901s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.242501333s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:37:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 6.242557009s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [.................................X........XX.X....X....X.XX.] duration: 3.614519001s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:37:34 UTC) orbiter warning: context deadline exceeded Duration: 3.614546492s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.543531417s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:37:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.543558589s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.545313462s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:36:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.545330344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [..................................X........XX.X....X....X.XX] duration: 5.001171045s trace: https://jaeger.dev.storj.io/trace/7b14c98be68176ca FAIL (2025-12-25 12:36:43 UTC) orbiter warning: context deadline exceeded Duration: 5.001195913s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.807079822s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:36:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.807106161s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 7 out of 60 is failed (threshold=6) [...................................X........XX.X....X....X.X] duration: 5.001803307s trace: https://jaeger.dev.storj.io/trace/17015cf567a86cf3 FAIL (2025-12-25 12:36:15 UTC) orbiter warning: context deadline exceeded Duration: 5.001825479s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.584191894s trace: https://jaeger.dev.storj.io/trace/2fdba245d87622ee FAIL (2025-12-25 12:35:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.376526002 (476h26m56.247609037s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377221506 (476h26m56.248304541s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.377449951 (476h26m56.248532986s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378207363 (476h26m56.249290398s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.378671797 (476h26m56.249754832s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379075266 (476h26m56.250158301s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.379741615 (476h26m56.25082465s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381452102 (476h26m56.252535137s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.381918359 (476h26m56.253001404s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.383615932 (476h26m56.254698967s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.384443127 (476h26m56.255526172s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.385393496 (476h26m56.256476542s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.386373122 (476h26m56.257456157s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:56.389571783 (476h26m56.260654828s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Uploading piece... Dec 25 11:55:57.852322963 (476h26m57.723405998s) [upload=418217]: Next returned nil segment Dec 25 11:55:57.852334997 (476h26m57.723418032s) [upload=418217]: Waiting for error group managing segments... Dec 25 11:55:58.250280581 (476h26m58.121363616s) [noise=true , address=145.79.227.40:4065 , piece_id=WVEBLZMESCWJX47XXEHZIKKCJI4V4T2SU2GGOZODCCTIWVSUFWYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.256878284 (476h26m58.127961329s) [noise=true , address=145.79.227.24:4593 , piece_id=C6GWB4XMSVPOPSNUHTOPUJSHLACGSN42AZEJU7VHCQYFT4THSUWQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.257509676 (476h26m58.128592721s) [noise=true , address=145.79.227.52:4479 , piece_id=IR52R3ZZU7BH5NL4C5OJAOF63E4SIJ3U6JR5Q73KCWNA4RMVCQTA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.269589378 (476h26m58.140672413s) [noise=true , address=145.79.227.3:4163 , piece_id=SAMETZN4HS5LEAB3H5DY7W5PS4HIADD47CN7DSD6ZUCLPWJK43IQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.276678868 (476h26m58.147761903s) [noise=true , address=145.79.227.16:4918 , piece_id=VVI3WNUW3FUBH777L7X42DWPJAIRT46XMP3BBSYQHQ2ZIHZ5XMYA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.450947134 (476h26m58.322030179s) [noise=true , address=77.90.31.21:4892 , piece_id=DHQFNPJCX3R6SRUELG4P4PXWFRMZIKO4E3BD6F4JAWHL4DCA2VIQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.454135244 (476h26m58.325218289s) [noise=true , address=77.90.31.11:4583 , piece_id=CJ6LUUFZU7M6O7OFHQAPPYY5XQ7DAKJUG52SE7ACATN5K3TGUN3A, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.500778761 (476h26m58.371861796s) [noise=true , address=45.91.116.6:4981 , piece_id=KRHMAMUOYO5GMABYW4QCHZOCXYPPJFEGM7ZEZ25XIVLZIGU3MREA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.508740220 (476h26m58.379823265s) [noise=true , address=45.91.116.8:4059 , piece_id=72TLFBLHC7QVNMCISTZSBRZ4P5RVY5GXFCFRJSK35N6X6LHFX4KQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.541777188 (476h26m58.412860233s) [noise=true , address=77.90.31.16:4762 , piece_id=TP7BJONAAMJX5EHBRAXVPFHKU2PHG7ZNR6L4VGSNHZR63KM7H4ZQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.558039314 (476h26m58.429122359s) [noise=true , address=45.91.116.13:4169 , piece_id=TM4KNYJSJJIUPKYGH2XUVLG4WZAQPSNXO4EHZL66TFQU524444VQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.560961519 (476h26m58.432044574s) [noise=true , address=77.90.31.12:4622 , piece_id=2X5J45RFURKMA25IZCQCAVKGS4ZBVYTAJ66GRIOIKOVKHSSED4VA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.583956886 (476h26m58.455039931s) [noise=true , address=77.90.31.13:4659 , piece_id=2RYCPQZYLB2SPQ64PSQXZXDXVKE4NHSHSCD37MY633BS6T6W7BRA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625863753 (476h26m58.496946798s) [noise=true , address=45.91.116.14:4193 , piece_id=26ETZYQKJX3ZTYRMBWURK2OM3SWOPSFXTGLGWWFJUYSEQI3RKAGA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: <nil> Dec 25 11:55:58.625887849 (476h26m58.496970894s) [seg_pos={0 0}, upload=418217]: Segment reached optimal threshold of 14 pieces. Dec 25 11:55:58.626098220 (476h26m58.497181285s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gczGyJMCtqUfaac5oy7o4CTwypVF8AZjrAAjoJweU3kBFbuXw): protocol: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47560->77.90.31.19:4867: use of closed network connection Dec 25 11:55:58.626107908 (476h26m58.497190953s) [noise=true , address=77.90.31.19:4867 , piece_id=Q2AU7B2P72GDQVOQAS7QGSZXG5NDJEVJFVGAENBR7PHRXTAQF2QA, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626333358 (476h26m58.497416423s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626341914 (476h26m58.497424959s) [noise=true , address=145.79.227.30:4792 , piece_id=Z3BM27BQNS4YJMAT5UVUCSJAJOT2HNV7GIPMPAACJPWJGCUI47QQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626470760 (476h26m58.497553825s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59650->64.49.13.169:4705: use of closed network connection Dec 25 11:55:58.626479766 (476h26m58.497562812s) [noise=true , address=64.49.13.169:4705 , piece_id=RHYOPZKJU2PPU3IPTJ54DTFMET3DHWFFCZRKHGN2HXMTLOV2NPBQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626644050 (476h26m58.497727115s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jjRZPZ6TDPepFAmvv81dREkDJDrXqeu9wqzNYkeaGemFFdxwK): protocol: expected piece hash; context canceled; EOF Dec 25 11:55:58.626653097 (476h26m58.497736152s) [noise=true , address=77.90.31.10:4546 , piece_id=EJ7LENBDRW73X2KCPLVF7NRGUR5M34CV7QTL4J27CEUOWC3DFFNQ, seg_pos={0 0}, upload=418217]: Long tail cancellation due to optimal threshold Dec 25 11:55:58.626672273 (476h26m58.497755318s) [seg_pos={0 0}, upload=418217]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 25 11:55:58.626762154 (476h26m58.497845209s) [upload=418217]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 11:55:58.626769067 (476h26m58.497852122s) [upload=418217]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 11:55:58.926594044 (476h26m58.797677089s) [upload=418217]: Done uploading segments... Dec 25 11:55:58.926619382 (476h26m58.797702427s) [upload=418217]: Upload finished. err: <nil> Duration: 5.584224696s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 6 out of 60 is failed (threshold=6) [....................................X........XX.X....X....X.] duration: 3.400098808s trace: https://jaeger.dev.storj.io/trace/325e1b1a3935fe2a FAIL (2025-12-25 12:35:06 UTC) orbiter warning: context deadline exceeded Duration: 3.400191294s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)