Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.910984167s FAIL (2025-06-28 12:47:49 UTC) orbiter warning: context deadline exceeded Duration: 5.911004515s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.231419094s FAIL (2025-06-28 12:47:10 UTC) orbiter warning: context deadline exceeded Duration: 6.231445314s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.09007136s FAIL (2025-06-28 12:46:05 UTC) orbiter warning: context deadline exceeded Duration: 6.09010743s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.93271255s FAIL (2025-06-28 12:45:10 UTC) orbiter warning: context deadline exceeded Duration: 5.932757656s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.860016379s FAIL (2025-06-28 12:44:38 UTC) orbiter warning: context deadline exceeded Duration: 5.860042358s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.043965861s FAIL (2025-06-28 12:44:20 UTC) orbiter warning: context deadline exceeded Duration: 6.043987382s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.951851104s FAIL (2025-06-28 12:43:36 UTC) orbiter warning: context deadline exceeded Duration: 5.951873876s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.081416059s FAIL (2025-06-28 12:43:00 UTC) orbiter warning: context deadline exceeded Duration: 6.081448832s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.951278888s FAIL (2025-06-28 12:41:46 UTC) orbiter warning: context deadline exceeded Duration: 5.951300418s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.062218031s FAIL (2025-06-28 12:41:09 UTC) orbiter warning: context deadline exceeded Duration: 6.062238118s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.026463405s FAIL (2025-06-28 12:39:55 UTC) orbiter warning: context deadline exceeded Duration: 6.026487251s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.366263713s FAIL (2025-06-28 12:39:30 UTC) orbiter warning: context deadline exceeded Duration: 6.366313728s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.99128181s FAIL (2025-06-28 12:38:40 UTC) orbiter warning: context deadline exceeded Duration: 5.991304944s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 6.236575043s FAIL (2025-06-28 12:37:41 UTC) orbiter warning: context deadline exceeded Duration: 6.236620251s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.984266549s FAIL (2025-06-28 12:37:19 UTC) orbiter warning: context deadline exceeded Duration: 5.984288661s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 6.184279578s FAIL (2025-06-28 12:36:39 UTC) orbiter warning: context deadline exceeded Duration: 6.184303984s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.067754662s FAIL (2025-06-28 12:35:34 UTC) orbiter warning: context deadline exceeded Duration: 6.0677748s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.023179173s FAIL (2025-06-28 12:34:21 UTC) orbiter warning: context deadline exceeded Duration: 6.023201985s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.898488692s FAIL (2025-06-28 12:33:33 UTC) orbiter warning: context deadline exceeded Duration: 5.898503119s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.300400849s FAIL (2025-06-28 12:32:39 UTC) orbiter warning: context deadline exceeded Duration: 6.300432058s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.976151706s FAIL (2025-06-28 12:31:50 UTC) orbiter warning: context deadline exceeded Duration: 5.976169389s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 6.006295546s FAIL (2025-06-28 12:31:25 UTC) orbiter warning: context deadline exceeded Duration: 6.006322226s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 6.224533238s FAIL (2025-06-28 12:30:46 UTC) orbiter warning: context deadline exceeded Duration: 6.224562114s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.113310001s FAIL (2025-06-28 12:30:07 UTC) orbiter warning: context deadline exceeded Duration: 6.113333717s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.060716806s FAIL (2025-06-28 12:29:00 UTC) orbiter warning: context deadline exceeded Duration: 6.060741033s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 6.24910911s FAIL (2025-06-28 12:28:36 UTC) orbiter warning: context deadline exceeded Duration: 6.249130391s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.172766571s FAIL (2025-06-28 12:27:27 UTC) orbiter warning: context deadline exceeded Duration: 6.172786149s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 6.209166526s FAIL (2025-06-28 12:26:46 UTC) orbiter warning: context deadline exceeded Duration: 6.209191423s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.218738067s FAIL (2025-06-28 12:26:24 UTC) orbiter warning: context deadline exceeded Duration: 6.218752666s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.048134677s FAIL (2025-06-28 12:26:02 UTC) orbiter warning: context deadline exceeded Duration: 6.048161808s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.933685764s FAIL (2025-06-28 12:25:31 UTC) orbiter warning: context deadline exceeded Duration: 5.933701574s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.035873367s FAIL (2025-06-28 12:24:52 UTC) orbiter warning: context deadline exceeded Duration: 6.035886903s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.150219562s FAIL (2025-06-28 12:23:51 UTC) orbiter warning: context deadline exceeded Duration: 6.150242145s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.900390488s FAIL (2025-06-28 12:22:45 UTC) orbiter warning: context deadline exceeded Duration: 5.90043896s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.066259571s FAIL (2025-06-28 12:22:22 UTC) orbiter warning: context deadline exceeded Duration: 6.066284739s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.112083726s FAIL (2025-06-28 12:21:44 UTC) orbiter warning: context deadline exceeded Duration: 6.112103253s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.974245583s FAIL (2025-06-28 12:20:51 UTC) orbiter warning: context deadline exceeded Duration: 5.974258799s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.842760912s FAIL (2025-06-28 12:20:01 UTC) orbiter warning: context deadline exceeded Duration: 5.842772575s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.00046421s FAIL (2025-06-28 12:19:55 UTC) orbiter warning: context deadline exceeded Duration: 30.000485552s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.756403611s FAIL (2025-06-28 04:49:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.756434501s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.6480942s FAIL (2025-06-28 04:48:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.648119438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.840862786s FAIL (2025-06-28 04:47:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.840884599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.701000028s FAIL (2025-06-28 04:47:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.701028322s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.680628743s FAIL (2025-06-28 04:46:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.680657267s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.851706901s FAIL (2025-06-28 04:45:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.851731418s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.68296925s FAIL (2025-06-28 04:45:05 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.682996842s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.664970716s FAIL (2025-06-28 04:44:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.664998069s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.662541703s FAIL (2025-06-28 04:43:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.662567081s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.771273051s FAIL (2025-06-28 04:42:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.771298208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.627618835s FAIL (2025-06-28 04:42:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== _SegmentCommit Jun 28 04:06:08.368760495 (114h55m43.139401831s) [upload=91894]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 28 04:06:08.668058331 (114h55m43.438699666s) [upload=91894]: Done uploading segments... Jun 28 04:06:08.668077908 (114h55m43.438719243s) [upload=91894]: Upload finished. err: <nil> Jun 28 04:06:28.245031102 (114h56m3.015672437s) [upload=91899]: Starting upload Jun 28 04:06:28.245091849 (114h56m3.015733174s) [upload=91899]: Done starting upload Jun 28 04:06:28.245179016 (114h56m3.015820351s) [upload=91899]: Uploading segments... Jun 28 04:06:28.245299867 (114h56m3.015941202s) [upload=91899]: Got next segment. Inline: false Jun 28 04:06:28.245326247 (114h56m3.015967582s) [upload=91899]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 28 04:06:28.245330746 (114h56m3.015972081s) [upload=91899]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 28 04:06:28.481966840 (114h56m3.252608195s) [seg_pos={0 0}, upload=91899]: Begin upload segment... Jun 28 04:06:28.482029781 (114h56m3.252671116s) [seg_pos={0 0}, upload=91899]: Done begin upload segment. Jun 28 04:06:28.482059808 (114h56m3.252701143s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482091159 (114h56m3.252732494s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482395551 (114h56m3.253036886s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482528094 (114h56m3.253169429s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482665136 (114h56m3.253306471s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482818480 (114h56m3.253459815s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.482895788 (114h56m3.253537123s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483074860 (114h56m3.253716205s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483257460 (114h56m3.253898795s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483334848 (114h56m3.253976183s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483523759 (114h56m3.254165104s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483619382 (114h56m3.254260717s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483750223 (114h56m3.254391558s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483927832 (114h56m3.254569167s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.483966917 (114h56m3.254608252s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:28.484237855 (114h56m3.25487919s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Uploading piece... Jun 28 04:06:30.087932713 (114h56m4.858574048s) [upload=91899]: Next returned nil segment Jun 28 04:06:30.087951258 (114h56m4.858592593s) [upload=91899]: Waiting for error group managing segments... Jun 28 04:06:30.414331358 (114h56m5.184972693s) [noise=true , address=104.36.82.12:4150 , piece_id=IRGRWGBPBSLMZWGZOGH67VQZEWQGSVEZGKHALJ5JOKLEZLOQLOXQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.433633187 (114h56m5.204274532s) [noise=true , address=104.36.82.16:4295 , piece_id=4EQTTHYVDDG3VFB5XPGGWQHLRQA6QQZ4J6UWTQWECJNJAWDORXGQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.434378623 (114h56m5.205019958s) [noise=true , address=104.36.82.14:4222 , piece_id=YKW5XTYL3THTHATRMHS3RR5HALMBIBIQUQZKPOWUMB2SDUBRYUFA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.445591745 (114h56m5.21623308s) [noise=true , address=104.36.82.15:4258 , piece_id=PCWP4JSBZT4B6GUBM6UDHSY5WHNO7VYPXRFQNAFUJV335GURH76A, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.464864056 (114h56m5.235505381s) [noise=true , address=104.36.82.11:4114 , piece_id=GVLY32G3J5DIKIQ26UQ3AZPWGBLQECBVVTIV4F347P6FCGUDQIQQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.483943808 (114h56m5.254585143s) [noise=true , address=104.36.82.4:4804 , piece_id=ZQK6EXCGOE2DTPEQJM6V4CQNE36TXGCCY36J3H5B5WPYYYR7E6DQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.541583844 (114h56m5.312225179s) [noise=true , address=45.91.116.9:5261 , piece_id=7XIRAED62VAHTGVYEMD4FY77CQOMMD24FRJ443U64BAWZMVUQUTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.542034617 (114h56m5.312675952s) [noise=true , address=45.91.116.10:4083 , piece_id=PR4DQ4UKH5E24G2IPAZXEP3R6EAZUFJ2FSYS3Z6LQ4FSKL2TLPRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.551772746 (114h56m5.322414081s) [noise=true , address=45.91.116.14:4190 , piece_id=JNHZJD3PM6KMXE4RWPIV3HTHBJT6JXJYIQJTWOENCSCPC23I27XA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.565503216 (114h56m5.336144551s) [noise=true , address=45.91.116.8:4048 , piece_id=PYQ7WLPQOF3DCY4R5HCNV32DK5PIQBN345MUOA4WWNCXYWRDHLTA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:30.666183975 (114h56m5.436825341s) [noise=true , address=45.91.116.7:4009 , piece_id=75RZ7A55YQ3N25EWQO5WPKCKIWSCB4XW7ZTQGUBADGAAQSNCYKRA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: <nil> Jun 28 04:06:57.603988300 (114h56m32.374629635s) [noise=true , address=77.90.31.11:4555 , piece_id=VFHE7TNWIHGLUYLNJGJKHSSP6MOHKMIAR6I36AQQHI43DG6D5QTQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, address:77.90.31.11:4555): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604142113 (114h56m32.374783449s) [noise=true , address=77.90.31.13:4626 , piece_id=5PHLCL763KD6XGCLNJDRAOBUFGY4IRYM2R5K3HTJNGJAUVS4YRUA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47782->77.90.31.13:4626: use of closed network connection Jun 28 04:06:57.604210886 (114h56m32.374852221s) [noise=true , address=77.90.31.21:4877 , piece_id=42C5Z24HFAN2RSDBNQMOHOQOZXWFCVXKETHZRTR7FOWBPFUPNQAQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40666->77.90.31.21:4877: use of closed network connection Jun 28 04:06:57.604216296 (114h56m32.374857631s) [noise=true , address=77.90.31.10:4518 , piece_id=VT4FPYRNJ3UFJT3ESCUFLQV3KBIFV2JLONJZW5IWJEUN6XIOFFZQ, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, address:77.90.31.10:4518): protocol: expected piece hash; context deadline exceeded; EOF Jun 28 04:06:57.604230052 (114h56m32.374871387s) [noise=true , address=77.90.31.14:4663 , piece_id=GTWRLAEQSANDKBHFUJW6HQ4FUMKNSD4B47PQ5WXGVBIUPBOLXSJA, seg_pos={0 0}, upload=91899]: Done uploading piece. err: ecclient: upload failed (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, address:77.90.31.14:4663): protocol: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35244->77.90.31.14:4663: use of closed network connection Jun 28 04:06:57.604245582 (114h56m32.374886907s) [seg_pos={0 0}, upload=91899]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jun 28 04:06:57.847138587 (114h56m32.617779912s) [upload=91899]: Done uploading segments... Jun 28 04:06:57.847164727 (114h56m32.617806062s) [upload=91899]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration:* 5.627645156s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)