Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 891057 reports (Page 11424 of 17822)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.679132616s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:25:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.679209552s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.524123781s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:24:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524139913s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.47924184s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:24:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.479269715s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.581562155s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:23:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.581583815s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.468305674s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:22:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.46833446s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.591125897s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:21:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.59115442s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.561991008s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:20:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.562016496s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.555700067s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:20:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.55572289s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.377767795s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:19:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.377845783s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.572505128s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:18:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572528262s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.635658814s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:17:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.635689543s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.669675519s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:16:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.669700065s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.639675452s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:15:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.639700769s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.629482904s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:14:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.629508032s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.659346499s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:13:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.659543003s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.60869611s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:13:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608725357s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.48861061s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:12:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.488637902s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.583886349s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:11:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.583915895s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.494634829s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:11:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.494662702s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.43683149s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:09:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.436863331s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.595321783s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:08:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.595349707s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.427393358s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:07:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.427418104s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.572633872s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:06:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572659641s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.500172262s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:06:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.500198972s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.564861392s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:05:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564884756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 3.547553626s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:05:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.547580829s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.572216157s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:04:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.572247997s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.578012866s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:04:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.578043162s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.538015524s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:03:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.538048826s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.625702784s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:02:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.62573211s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001661791s trace: https://jaeger.dev.storj.io/trace/771f5a60534b2000 FAIL (2025-12-23 20:01:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001680296s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.850681593s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:33:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.85070633s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.901248542s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:32:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.901272487s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.882802696s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:31:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.882835639s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.783373676s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:31:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.783401841s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.916221908s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:30:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.91625446s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.726149978s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:30:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.72617725s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 6.59132909s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:29:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 6.591368435s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.956399806s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:28:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.956428129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.992428229s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:27:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.99248689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.852444124s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:26:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.852475294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.659283837s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:25:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 6.659311431s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 15.385625424s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:25:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 15.385666152s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.965349144s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:25:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.965371499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 11.08028252s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:24:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 11.080315253s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 7.160998569s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:24:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 7.161044307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.708751308s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:24:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.708840379s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 9.63779977s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:23:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 9.637828657s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.911966736s trace: https://jaeger.dev.storj.io/trace/1de3548b82915ef FAIL (2025-12-23 18:23:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:43:48.086744421 (434h14m47.957827456s) [upload=381190]: Starting upload Dec 23 17:43:48.086814404 (434h14m47.957897459s) [upload=381190]: Done starting upload Dec 23 17:43:48.086927901 (434h14m47.958010936s) [upload=381190]: Uploading segments... Dec 23 17:43:48.087052517 (434h14m47.958135562s) [upload=381190]: Got next segment. Inline: false Dec 23 17:43:48.087110508 (434h14m47.958193543s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:43:48.087116910 (434h14m47.958199945s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:43:48.598667297 (434h14m48.469750332s) [seg_pos={0 0}, upload=381190]: Begin upload segment... Dec 23 17:43:48.598711030 (434h14m48.469794075s) [seg_pos={0 0}, upload=381190]: Done begin upload segment. Dec 23 17:43:48.598740587 (434h14m48.469823632s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.598787195 (434h14m48.46987023s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599218216 (434h14m48.470301252s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599349726 (434h14m48.470432761s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599645169 (434h14m48.470728214s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.599958305 (434h14m48.47104134s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600224121 (434h14m48.471307156s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.600490108 (434h14m48.471573143s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602211246 (434h14m48.473294281s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.602660240 (434h14m48.473743315s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.603040614 (434h14m48.474123649s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604349807 (434h14m48.475432843s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.604674265 (434h14m48.47575731s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.605975103 (434h14m48.477058148s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.607777614 (434h14m48.478860649s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:48.608796235 (434h14m48.47987928s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Uploading piece... Dec 23 17:43:50.493160293 (434h14m50.364243328s) [upload=381190]: Next returned nil segment Dec 23 17:43:50.493176243 (434h14m50.364259288s) [upload=381190]: Waiting for error group managing segments... Dec 23 17:43:50.823835453 (434h14m50.694918488s) [noise=true , address=77.90.31.10:4519 , piece_id=YYV7MJMXJ2GNRMJMOWSKFZZ3KNSC5KU5JVKSQ7QMD5QM7C763N3A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.878247713 (434h14m50.749330748s) [noise=true , address=45.91.116.13:4154 , piece_id=VGPA2C74KZYXRX35A4325EK3EJ6DVELHX5RCSEBRBZBOHWTCLVPA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.936445201 (434h14m50.807528246s) [noise=true , address=45.91.116.12:4228 , piece_id=JHKZC6CVLX7DIQPDN2WQGNGWRJVRAZ2WOSBPPDGNLFA7L2VHD54A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:50.955217256 (434h14m50.826300301s) [noise=true , address=45.91.116.4:4939 , piece_id=SJSOPCNVS2TCXUOG2AAYNRWLZTTGS2UTA6PGFPURZ6WGSVJ6OCUA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.654729361 (434h14m51.525812406s) [noise=true , address=145.79.227.9:4367 , piece_id=KHEYRHIKEUU3K2CKVX262KEFCXJ5TM73ZSC7WCIW3D7SCXXPXH6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.749137570 (434h14m51.620220605s) [noise=true , address=145.79.227.45:4231 , piece_id=GHEQUC2VYC3TH6HLFDZQENXLAECLTIAWHCBS22C5P74PX4WB7BMA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.775735577 (434h14m51.646818612s) [noise=true , address=45.91.116.8:4047 , piece_id=Z5F242RATMD34NERFHL6OAGLT3KKZKAIZR7BL6N7SNVHZMGCBSBA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.823106970 (434h14m51.694190015s) [noise=true , address=145.79.227.46:4603 , piece_id=WPNCVJGVRMQAQ3KQKJPF72U5GXSXKOF4HXAELUTHRZW7ZIACWNNA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:51.906987172 (434h14m51.778070207s) [noise=true , address=145.79.227.14:4840 , piece_id=TSHDD7C4VVVSETI5CNINNWZ54OUO3QVX7VSJ5UMXJGLRRETDBMKQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:52.556137961 (434h14m52.427220996s) [noise=true , address=145.79.227.51:4409 , piece_id=MNZZDEQMXP42VOMDITMBU2YFS3VAUVQCQ5647XFOGIB6S2NZILTQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:53.415729905 (434h14m53.286812941s) [noise=true , address=145.79.227.4:4185 , piece_id=TRZJBNZHVOZGPSNOIOPSGRICXWBYPEVMGHIDVB2J4ZEO2OPLFBPQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.624680344 (434h14m56.495763389s) [noise=true , address=145.79.227.48:4301 , piece_id=LRGJPI54AYV4RX6RP3CYPLBGCRVJ2JDDSM2ILPJXH3DEWM7Q6ZSA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756552430 (434h14m56.627635475s) [noise=true , address=145.79.227.37:4567 , piece_id=XZMQ3K6ZPIHJZ2ZVGJWGZPH263KHMNJYZVU3V2DX4UKM77FMSB4A, seg_pos={0 0}, upload=381190]: Done uploading piece. err: <nil> Dec 23 17:43:56.756572950 (434h14m56.627655985s) [seg_pos={0 0}, upload=381190]: Segment reached optimal threshold of 13 pieces. Dec 23 17:43:56.756796545 (434h14m56.62787958s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QRixhWaUwXPdjcWKYHbiknME5JVomWYABY6egDD6VKwaTPB8K): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756802767 (434h14m56.627885802s) [noise=true , address=145.79.227.50:4374 , piece_id=3A7Q24UMO5U354T7DEUBMYBJOXVBMJUTPZDCOR6BY4YCGU72KTGA, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756888390 (434h14m56.627971435s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: read tcp 188.166.225.170:42476->145.79.227.40:4049: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 23 17:43:56.756903028 (434h14m56.627986073s) [noise=true , address=145.79.227.40:4049 , piece_id=NJGBVISHVW6JAXB3X3EY4QHPJZ72OKDIHSRUBLXTCNQXGZW5HK6Q, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.756987488 (434h14m56.628070523s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BfeGD2MGRPqsBRGLioTiTAcPykpSTUUx2CBVSPQcz4LQqFdWR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:56.756993751 (434h14m56.628076786s) [noise=true , address=145.79.227.41:4087 , piece_id=SGI44QI7FVVKXIFCH6S66CAHUGFQFZ7RAPT3DYTTPOLO3JDDRIEQ, seg_pos={0 0}, upload=381190]: Long tail cancellation due to optimal threshold Dec 23 17:43:56.757006775 (434h14m56.62808981s) [seg_pos={0 0}, upload=381190]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:43:56.757066239 (434h14m56.628149274s) [upload=381190]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:56.757070457 (434h14m56.628153492s) [upload=381190]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:44:02.390007212 (434h15m2.261090267s) [upload=381190]: Done uploading segments... Dec 23 17:44:02.390034304 (434h15m2.261117349s) [upload=381190]: Upload finished. err: <nil> Duration: 5.912004007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 7.440904711s trace: https://jaeger.dev.storj.io/trace/2d8499561e832555 FAIL (2025-12-23 18:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694150953 (434h14m17.565233998s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694524234 (434h14m17.565607279s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.694916100 (434h14m17.565999145s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695317846 (434h14m17.566400881s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695644056 (434h14m17.566727102s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.695935591 (434h14m17.567018636s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696198001 (434h14m17.567281046s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696464760 (434h14m17.567547795s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.696725065 (434h14m17.5678081s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.697000780 (434h14m17.568083815s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698381980 (434h14m17.569465025s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.698780889 (434h14m17.569863935s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699072395 (434h14m17.57015544s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.699756207 (434h14m17.570839242s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:17.700449918 (434h14m17.571532973s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Uploading piece... Dec 23 17:43:18.996789420 (434h14m18.867872475s) [upload=381181]: Next returned nil segment Dec 23 17:43:18.996806513 (434h14m18.867889558s) [upload=381181]: Waiting for error group managing segments... Dec 23 17:43:19.337632251 (434h14m19.208715287s) [noise=true , address=64.49.13.137:4501 , piece_id=Z57D6P6AP6ADOXCCPJQNB5BSFDGARL6J2UCKLL6A3CT3QDZNKVCA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.351474710 (434h14m19.222557745s) [noise=true , address=64.49.13.170:4728 , piece_id=LJQ36R22K5CTLKBFPARL4PKRX46C55R45U4FV5LWHBWQU7JRMVUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.777150228 (434h14m19.648233263s) [noise=true , address=145.79.227.31:4815 , piece_id=QNUR7VPT5I7IGQTT4RGJZBPGAU5TH3NIDR6QDKVR34E3Y2SMMNJQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.786259823 (434h14m19.657342868s) [noise=true , address=77.90.31.10:4533 , piece_id=KXMM5QDZRBM4THP4LLDFJ2SBUXTJL52SFFB7CKB4DMYWIQNTPZJA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.789005721 (434h14m19.660088766s) [noise=true , address=77.90.31.19:4873 , piece_id=VCJOFTMYNLZGBYHV3F35EMWICMAI65W4SOZHXT65S326WCHAKJ7Q, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.809981934 (434h14m19.681064969s) [noise=true , address=77.90.31.11:4565 , piece_id=D3C7E7MF5OIZIPFVT3ARKOESGBLH6WUGVQAGUPMW5VPSFPFH6FYA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.860166353 (434h14m19.731249388s) [noise=true , address=45.91.116.12:4252 , piece_id=SCIG56B3I4BIQGBAAUSOVDSQYCJFSCJTXD5O2E5BZS5NHIH5CAUQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.870306351 (434h14m19.741389386s) [noise=true , address=45.91.116.6:4993 , piece_id=EVQXSJIL5C7A6N2C2DST5HFAJMYZLGSPMCKML2F6556WPTPU67RA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.875103586 (434h14m19.746186631s) [noise=true , address=45.91.116.8:4058 , piece_id=B662HA7UBEULOTEELCD2YZ4LMS7KGBOEBEVMVY35GFWU65RHO3TA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.902508389 (434h14m19.773591424s) [noise=true , address=45.91.116.11:4129 , piece_id=GASTPPMR2ZPBF5YD3THNYNTZFVYR3H2WBQ7DEKYCMB5CZBG4RCEA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:19.917888896 (434h14m19.788971931s) [noise=true , address=45.91.116.9:4278 , piece_id=MMM6WXHOZZQCTJFBDBYQOJH7UVXSZDEFWBNDDEPJKUFPOTYF3V3A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.410588767 (434h14m20.281671802s) [noise=true , address=145.79.227.48:4312 , piece_id=LVIND7H22EPSKLZIFTGXTNLJTIZQMYL3DBDTB6QE7ZL3PRRHSXRQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.777502069 (434h14m20.648585104s) [noise=true , address=145.79.227.21:4491 , piece_id=5PSBP2JDCI3CLEKXBWXXBMMEJVTXI3TGGKON33LC6CUF3A4JLM7A, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971652354 (434h14m20.842735389s) [noise=true , address=145.79.227.44:4201 , piece_id=7TOHMQXUUCUYVLUQCYHZWZLCEVNKGMACVEESD2AFXQDIVYFRYMDQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: <nil> Dec 23 17:43:20.971672252 (434h14m20.842755287s) [seg_pos={0 0}, upload=381181]: Segment reached optimal threshold of 14 pieces. Dec 23 17:43:20.972461524 (434h14m20.843544559s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aR2XEwckLo7ufnCtsmyG2PS3vNiqpZs9b66UhuH7vUixcePS1): protocol: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40492->145.79.227.17:4974: use of closed network connection Dec 23 17:43:20.972475351 (434h14m20.843558386s) [noise=true , address=145.79.227.17:4974 , piece_id=N7LPOB6FOF6YSVHDTEWU26CPCWJSVILDV3GVDFGJWZCWL6GWDFVA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972696051 (434h14m20.843779086s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:182cMUjmC3Ku6z4KucQpqTS8NcKz7VJKeXt9tFYPnUtDUj6A2w): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972702173 (434h14m20.843785208s) [noise=true , address=145.79.227.16:4919 , piece_id=R76XSE2BKBQNNPIIZYWTZODRIJSJDS7TAHBZMXJ5MSLJQTSNL7CA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.972885402 (434h14m20.843968437s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ts2cPQYUxdscD7RBpVxiHk36KAtJnXnAP9FwY7gCpBXHeS1Cg): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.972897665 (434h14m20.8439807s) [noise=true , address=145.79.227.18:4021 , piece_id=JLTZR4ZMP3S5JI35EXOQAHMONQGCVELK5PJK32VBC5JNTNTCSNOA, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973157299 (434h14m20.844240334s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Gbrbe3SdK7ccTugeFtmHE7gGWmLrZrGRXvb47EbiJPjfZfbKR): protocol: expected piece hash; context canceled; EOF Dec 23 17:43:20.973170465 (434h14m20.8442535s) [noise=true , address=145.79.227.44:4215 , piece_id=CMUJHLMH3QGPMQFIZT5DWNSTF6IWXFRIUPSC6FFCJNC4O6BU2RBQ, seg_pos={0 0}, upload=381181]: Long tail cancellation due to optimal threshold Dec 23 17:43:20.973197887 (434h14m20.844280922s) [seg_pos={0 0}, upload=381181]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 23 17:43:20.973332623 (434h14m20.844415658s) [upload=381181]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:43:20.973350327 (434h14m20.844433362s) [upload=381181]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:43:21.278516423 (434h14m21.149599478s) [upload=381181]: Done uploading segments... Dec 23 17:43:21.278541330 (434h14m21.149624365s) [upload=381181]: Upload finished. err: <nil> Duration: 7.440933896s core uplink us-select-1 us1-placement-20

1
...
11422
11423
11424
11425
11426
...
17822