Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.665327201s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:37:14 UTC) orbiter warning: context deadline exceeded Duration: 3.665350265s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 4.58203672s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:36:11 UTC) orbiter warning: context deadline exceeded Duration: 4.582061488s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.532401591s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:35:21 UTC) orbiter warning: context deadline exceeded Duration: 3.532419535s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................X..........................] duration: 3.548891744s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:34:14 UTC) orbiter warning: context deadline exceeded Duration: 3.548916019s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................X.........................] duration: 3.606535934s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:33:01 UTC) orbiter warning: context deadline exceeded Duration: 3.60656038s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................X........................] duration: 3.508066135s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:31:55 UTC) orbiter warning: context deadline exceeded Duration: 3.508124817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................X.......................] duration: 3.776764517s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:31:12 UTC) orbiter warning: context deadline exceeded Duration: 3.776794845s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................X......................] duration: 3.702759188s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:30:03 UTC) orbiter warning: context deadline exceeded Duration: 3.702789725s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................X.....................] duration: 3.446706646s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:29:41 UTC) orbiter warning: context deadline exceeded Duration: 3.446737014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................X....................] duration: 3.676507682s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:28:35 UTC) orbiter warning: context deadline exceeded Duration: 3.67653745s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................X...................] duration: 3.517618233s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:27:48 UTC) orbiter warning: context deadline exceeded Duration: 3.517647189s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................X..................] duration: 3.493827052s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:26:55 UTC) orbiter warning: context deadline exceeded Duration: 3.493858611s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 3.649956004s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:26:39 UTC) orbiter warning: context deadline exceeded Duration: 3.649974188s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 3.595459653s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:26:17 UTC) orbiter warning: context deadline exceeded Duration: 3.595492013s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 3.631977736s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:25:15 UTC) orbiter warning: context deadline exceeded Duration: 3.632000458s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 3.479908161s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:24:40 UTC) orbiter warning: context deadline exceeded Duration: 3.479936817s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 3.511910741s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:24:14 UTC) orbiter warning: context deadline exceeded Duration: 3.511933894s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 3.547496368s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:23:05 UTC) orbiter warning: context deadline exceeded Duration: 3.54753918s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 3.613995826s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:22:21 UTC) orbiter warning: context deadline exceeded Duration: 3.614021756s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 3.495964362s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:21:22 UTC) orbiter warning: context deadline exceeded Duration: 3.495990682s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 3.543535525s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:20:44 UTC) orbiter warning: context deadline exceeded Duration: 3.543576855s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 3.535182553s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:19:42 UTC) orbiter warning: context deadline exceeded Duration: 3.535209393s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 3.462177279s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:19:04 UTC) orbiter warning: context deadline exceeded Duration: 3.462200332s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 3.790404062s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:18:21 UTC) orbiter warning: context deadline exceeded Duration: 3.790432497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 4.425977632s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:17:34 UTC) orbiter warning: context deadline exceeded Duration: 4.426007891s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 3.558918418s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:17:12 UTC) orbiter warning: context deadline exceeded Duration: 3.558937434s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 3.633881337s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:16:36 UTC) orbiter warning: context deadline exceeded Duration: 3.633908328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 3.567768421s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:15:43 UTC) orbiter warning: context deadline exceeded Duration: 3.567785674s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.629828593s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:14:45 UTC) orbiter warning: context deadline exceeded Duration: 3.629852728s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001400467s trace: https://jaeger.dev.storj.io/trace/7f520a989b7ec425 FAIL (2025-12-21 21:14:11 UTC) orbiter warning: context deadline exceeded Duration: 5.00142801s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.838402392s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 21:02:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.838425666s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.864769482s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 21:01:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.864852439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.873880019s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 21:00:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.873914254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.699644013s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 21:00:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.699669993s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.781724829s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:59:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.781749257s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.05859733s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:58:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.058617018s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.845682627s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.845709068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.685376636s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:57:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.685402055s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.979302951s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:55:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.97933451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.793576217s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:55:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.793607034s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.867421482s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:54:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.867449486s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.772338648s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:53:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.772368656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.809543723s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:52:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.809577256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.8485373s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.848574611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.121497198s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:51:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 6.121527335s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.836582177s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:50:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.836610442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 7.400333471s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:49:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 7.400386301s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.752581817s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:48:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.75261497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 15.016406081s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:47:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 15.016442409s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.927226929s trace: https://jaeger.dev.storj.io/trace/31c2c0194ef48da8 FAIL (2025-12-21 20:46:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 21 20:18:32.445345332 (388h49m32.316428397s) [upload=341279]: Starting upload Dec 21 20:18:32.445489336 (388h49m32.316572381s) [upload=341279]: Done starting upload Dec 21 20:18:32.445606230 (388h49m32.316689295s) [upload=341279]: Uploading segments... Dec 21 20:18:32.446058280 (388h49m32.317141315s) [upload=341279]: Got next segment. Inline: false Dec 21 20:18:32.446142190 (388h49m32.317225225s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 20:18:32.446149333 (388h49m32.317232378s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 20:18:32.699088159 (388h49m32.570171194s) [seg_pos={0 0}, upload=341279]: Begin upload segment... Dec 21 20:18:32.699158824 (388h49m32.570241859s) [seg_pos={0 0}, upload=341279]: Done begin upload segment. Dec 21 20:18:32.699207226 (388h49m32.570290262s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699262130 (388h49m32.570345176s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699344788 (388h49m32.570427834s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699769737 (388h49m32.570852773s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.699931064 (388h49m32.571014109s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700169067 (388h49m32.571252112s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700289517 (388h49m32.571372562s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700640235 (388h49m32.57172328s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.700725006 (388h49m32.571808041s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701063571 (388h49m32.572146606s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701084291 (388h49m32.572167326s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701413918 (388h49m32.572496953s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701498249 (388h49m32.572581314s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.701730571 (388h49m32.572813616s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.702205806 (388h49m32.573288911s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:32.705620438 (388h49m32.576703483s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Uploading piece... Dec 21 20:18:34.401105305 (388h49m34.27218835s) [upload=341279]: Next returned nil segment Dec 21 20:18:34.401121025 (388h49m34.27220407s) [upload=341279]: Waiting for error group managing segments... Dec 21 20:18:34.672829292 (388h49m34.543912327s) [noise=true , address=145.79.227.11:4731 , piece_id=CRSJHZXK7MXBPFZP4G6GEUBNLZG3GZ4C3UQ7SHJUOSORK4HLUK5Q, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682473907 (388h49m34.553556942s) [noise=true , address=145.79.227.38:4978 , piece_id=HWOBBQHGVKJAWMVXZ4F46JIOTMJPOYLUGALZ5SGYQPYNS6V33X5A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.682957398 (388h49m34.554040433s) [noise=true , address=145.79.227.45:4231 , piece_id=QHS2GMFZYSWWUMWPDTMT75CU3IDM5P4C3WUARDA3I7OUII2D64VA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.683338644 (388h49m34.554421679s) [noise=true , address=145.79.227.43:4158 , piece_id=KEA5A6CVQXMV2RYGPDRSMF3CEOQN6AIC3ZAOJHFJ6GU7EIJRZQLQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.704059039 (388h49m34.575142074s) [noise=true , address=145.79.227.37:4567 , piece_id=QRAJ2AJB3LLLABVLEKRSDEVENGNCW5QRCYP5YCUUGSNTAMIJ54CA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.740950405 (388h49m34.61203345s) [noise=true , address=145.79.227.42:4123 , piece_id=DIBHB2B46BFAE7WDRNQSFGQXHR5PGT5HKQ4AYBVY4AT7M2WSJMRQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.753235558 (388h49m34.624318594s) [noise=true , address=145.79.227.39:4014 , piece_id=AKEYMC3RZY3KEMNPZU7AUX6L34KVCR5PAWQMZHRKRVZGO4DWLR6A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.764021868 (388h49m34.635104903s) [noise=true , address=145.79.227.18:4006 , piece_id=ONPVDGAM7SQ5AMYWAM6TN6VJAUJMH7NMPNGLVDWLCFTE4Q5QMRDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.886907671 (388h49m34.757990716s) [noise=true , address=45.91.116.13:4155 , piece_id=QLAQR2WS6XGEURWNKVYQ7DRST4AZAI56K3NYURPVHLQSRXAP2AYA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.887876206 (388h49m34.758959241s) [noise=true , address=77.90.31.21:4880 , piece_id=BD4IFTLZD76RM7BPN3THABL4NURE2TPWYLVGHFWAPXAM7ROZRVDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.935331779 (388h49m34.806414814s) [noise=true , address=77.90.31.13:4627 , piece_id=52DP7OHSRLXGZIGDCB7KZT4QMR5JG7OCSH5V3ZJYKIC3SKVO4EYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.953989056 (388h49m34.825072091s) [noise=true , address=45.91.116.4:4937 , piece_id=SII2D4NSCKVKZVNIXSZJDBTTLSF4AONOUOY4MP5PIYQW34WPUHAA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964580153 (388h49m34.835663188s) [noise=true , address=45.91.116.14:4190 , piece_id=U2LQMDKPRTCZNKBGWRGRR7LXT2LU3L5PK5SMTNDEGO4JLPH27HYQ, seg_pos={0 0}, upload=341279]: Done uploading piece. err: <nil> Dec 21 20:18:34.964600741 (388h49m34.835683777s) [seg_pos={0 0}, upload=341279]: Segment reached optimal threshold of 13 pieces. Dec 21 20:18:34.964847100 (388h49m34.835930136s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964855597 (388h49m34.835938632s) [noise=true , address=45.91.116.8:4047 , piece_id=KYGYMIWUZ5LUUP3TENNQSED5GIPUZUJC7CYMV6MHN5T7O2OHRIDA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.964911824 (388h49m34.835994869s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 21 20:18:34.964921472 (388h49m34.836004507s) [noise=true , address=77.90.31.11:4556 , piece_id=XTB6XOHNYMN2EEIFHKGPU34SFWRNKN7PBOI2YI3RHPDZ4JXKJW4A, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965241402 (388h49m34.836324437s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: read tcp 188.166.225.170:43214->77.90.31.16:4736: use of closed network connection; context canceled Dec 21 20:18:34.965249507 (388h49m34.836332542s) [noise=true , address=77.90.31.16:4736 , piece_id=57FIT3O6B43ZPFNJJWD4JI4NUME6JCHA47EOTE6SEQMPY5C44TNA, seg_pos={0 0}, upload=341279]: Long tail cancellation due to optimal threshold Dec 21 20:18:34.965293290 (388h49m34.836376325s) [seg_pos={0 0}, upload=341279]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 20:18:34.965370958 (388h49m34.836454003s) [upload=341279]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 20:18:34.965376539 (388h49m34.836459595s) [upload=341279]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 20:18:35.249645163 (388h49m35.120728198s) [upload=341279]: Done uploading segments... Dec 21 20:18:35.249665261 (388h49m35.120748306s) [upload=341279]: Upload finished. err: <nil> Duration: 5.927250594s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)