Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.823793731s trace: https://jaeger.dev.storj.io/trace/2401c4f0c63f62dd FAIL (2025-12-16 19:59:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:53224->45.38.24.2:443: read: connection reset by peer Duration: 5.823822497s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.972401801s trace: https://jaeger.dev.storj.io/trace/2401c4f0c63f62dd FAIL (2025-12-16 19:58:38 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:53224->45.38.24.2:443: read: connection reset by peer Duration: 5.972440664s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.88261385s trace: https://jaeger.dev.storj.io/trace/2401c4f0c63f62dd FAIL (2025-12-16 19:58:23 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:53224->45.38.24.2:443: read: connection reset by peer Duration: 5.882631333s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 450.728513ms trace: https://jaeger.dev.storj.io/trace/2401c4f0c63f62dd FAIL (2025-12-16 19:57:15 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:53224->45.38.24.2:443: read: connection reset by peer Duration: 450.755644ms edge gateway us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 707.915215ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:33:35 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 707.940873ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 744.892678ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:33:06 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 744.951769ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 741.142847ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:32:34 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 741.164789ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 721.751282ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:31:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 721.789103ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 714.950342ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:30:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 714.978867ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 746.571616ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:30:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 746.602615ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 732.501532ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:30:05 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 732.559253ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 740.398581ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:29:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 740.438497ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 719.037535ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:28:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 719.070568ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 714.683363ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:28:00 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 714.708932ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 717.611849ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:27:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 717.641837ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 712.190155ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:26:32 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 712.223839ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 734.993467ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:25:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 735.022964ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 800.526705ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:25:01 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 800.550951ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 748.675271ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:24:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 748.721658ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 763.705972ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:23:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 763.741359ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 742.974003ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:22:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 742.993561ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 738.754428ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:21:39 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 738.778415ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 737.649816ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:20:26 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 737.755136ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 765.601631ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:19:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 765.623262ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 803.799871ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:18:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 803.826812ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 759.619074ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:17:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 759.63842ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 756.699563ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:17:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 756.724941ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 783.77414ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:16:29 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 783.800159ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 770.286678ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:16:11 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 770.33046ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.922143834s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:15:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.922181257s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 769.300429ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:15:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 769.331428ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.839079766s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:14:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.839110154s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 739.828468ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:14:31 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 739.851513ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.822632767s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:14:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.822664578s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 782.31421ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:13:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 782.355059ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 776.689016ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:13:22 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 776.719014ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.951011514s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:13:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.951043726s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 716.541016ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:12:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 716.572506ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.143824496s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:12:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 6.143848983s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.065306042s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:12:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 6.065335548s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 774.470151ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:11:51 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 774.503102ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.908100774s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:11:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.908132615s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 750.582736ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:11:17 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 750.611902ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.995873209s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:11:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.995898636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.943431103s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:10:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 5.943475127s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 721.792987ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:10:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 721.830748ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.003624752s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:10:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 6.003653286s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 748.596756ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:09:24 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 748.625199ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.278240438s trace: https://jaeger.dev.storj.io/trace/29bccf9eb32a68ab FAIL (2025-12-16 19:09:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ) context deadline exceeded; piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA) context deadline exceeded ======LOGS====== ments... Dec 16 18:27:30.461024428 (266h58m30.332107473s) [upload=234016]: Got next segment. Inline: false Dec 16 18:27:30.461057771 (266h58m30.332140806s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 16 18:27:30.461062530 (266h58m30.332145565s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 16 18:27:30.713043350 (266h58m30.584126385s) [seg_pos={0 0}, upload=234016]: Begin upload segment... Dec 16 18:27:30.713150904 (266h58m30.584233939s) [seg_pos={0 0}, upload=234016]: Done begin upload segment. Dec 16 18:27:30.713234162 (266h58m30.584317208s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713241937 (266h58m30.584324982s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.713727422 (266h58m30.584810457s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.714179613 (266h58m30.585262648s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715044219 (266h58m30.586127254s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715456414 (266h58m30.586539449s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.715769530 (266h58m30.586852576s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.716178550 (266h58m30.587261585s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717496459 (266h58m30.588579494s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.717519573 (266h58m30.588602608s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.718677177 (266h58m30.589760222s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719204942 (266h58m30.590287987s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.719856522 (266h58m30.590939567s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.721159023 (266h58m30.592242058s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.722473727 (266h58m30.593556762s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:30.724337565 (266h58m30.59542061s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Uploading piece... Dec 16 18:27:32.384365980 (266h58m32.255449015s) [upload=234016]: Next returned nil segment Dec 16 18:27:32.384388373 (266h58m32.255471408s) [upload=234016]: Waiting for error group managing segments... Dec 16 18:27:32.716227897 (266h58m32.587310932s) [noise=true , address=145.79.227.45:4231 , piece_id=ALPOXMSOJWIU3AL473NMALMW6TR34FL5IO2LBK2VJQFPZZW6OFZQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.720034635 (266h58m32.59111767s) [noise=true , address=145.79.227.47:4266 , piece_id=V5YZWY6YEVSUN5FUILWDKEIH4KJF7B5MLNH2FYLHQANQQ4JNRD3Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.740967836 (266h58m32.612050871s) [noise=true , address=145.79.227.18:4007 , piece_id=YBC6PPD3TCKSDX6D67CSWCETPYB7EY4DV44GV4G5HYMOMYINQU5Q, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.754359376 (266h58m32.625442411s) [noise=true , address=145.79.227.42:4122 , piece_id=VJRIB3YU2EE5GJ6HTQLCAMJBPEWSDJTY4CBRKIT2RXH6CMQIFDBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757130123 (266h58m32.628213168s) [noise=true , address=145.79.227.36:4517 , piece_id=SR7UETJ6SEFPWCOHBVU4QVHLWXKVFC3BZXP3O4UPWHFUWR2ZT3UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.757666203 (266h58m32.628749248s) [noise=true , address=145.79.227.21:4475 , piece_id=532SFDCBQ3BFWOL2TKISFEDNU5OY5OD4Z77HZXBCXFCY66OMMPBQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.765561507 (266h58m32.636644542s) [noise=true , address=145.79.227.29:4942 , piece_id=CI4I7PG3BZD5A6OKQ2AXPJQG7VV4GQGU6ZRGGIQOXQMEUDLZYIXA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.769584397 (266h58m32.640667432s) [noise=true , address=145.79.227.48:4301 , piece_id=QWINR4HHP4C5FSYUUSCJT3GKY4I47YX5YPUCSE6MVPBPJ65ZV2UQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.793925197 (266h58m32.665008232s) [noise=true , address=145.79.227.49:4338 , piece_id=NFJU2JSJXCXVPXT6GWQ7FRXHNQM42O7QHOYTXCW4EF3JGHR3RN7A, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.797255719 (266h58m32.668338754s) [noise=true , address=145.79.227.51:4409 , piece_id=JHRA5QUWKYOWLJ4GJAPYRHFARYI7ER66RFZSR2SBVGWWL5NUWVKQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.804265907 (266h58m32.675348952s) [noise=true , address=145.79.227.23:4545 , piece_id=5ZHH2Y6DSXJQS465SVQCXWVN4R6LYMOPIQ45DU5KMX2ZAVEL54QQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:32.976138625 (266h58m32.84722167s) [noise=true , address=45.91.116.10:4081 , piece_id=DG6BYH2FS3HGKKZ2YX3WVIN77QZ6HWVA6UTVJIHZKTQTY5NWMLQA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004834939 (266h58m32.875917984s) [noise=true , address=77.90.31.12:4590 , piece_id=PIERFD7I7FICIDXTUNN5QCZEVCX6AMMMTHF7AQUONAR75SGWPDSQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: <nil> Dec 16 18:27:33.004856729 (266h58m32.875939775s) [seg_pos={0 0}, upload=234016]: Segment reached optimal threshold of 13 pieces. Dec 16 18:27:33.005114411 (266h58m32.876197446s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 16 18:27:33.005122596 (266h58m32.876205641s) [noise=true , address=45.91.116.4:4940 , piece_id=I2Z35S7FQYFXCJR4XTMWOEDPZAWQKDRY4TAOWXVIK6BSPQ3MKDTA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005440571 (266h58m32.876523607s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38854->45.91.116.11:4117: use of closed network connection Dec 16 18:27:33.005448607 (266h58m32.876531652s) [noise=true , address=45.91.116.11:4117 , piece_id=TCJQYNDNREPUWTG3DB32JBNFWVWPT52M2SQGUVDKIRFRQIYFJWEA, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005539250 (266h58m32.876622295s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35860->77.90.31.19:4842: use of closed network connection Dec 16 18:27:33.005546503 (266h58m32.876629538s) [noise=true , address=77.90.31.19:4842 , piece_id=33D67VTE2RM64DPIFQUADQAN6TB5E4N6PUTAVZGR7OCAWLSABUTQ, seg_pos={0 0}, upload=234016]: Long tail cancellation due to optimal threshold Dec 16 18:27:33.005562484 (266h58m32.876645519s) [seg_pos={0 0}, upload=234016]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 16 18:27:33.005628159 (266h58m32.876711194s) [upload=234016]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 16 18:27:33.005632718 (266h58m32.876715763s) [upload=234016]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 16 18:27:33.310159814 (266h58m33.181242859s) [upload=234016]: Done uploading segments... Dec 16 18:27:33.310187557 (266h58m33.181270602s) [upload=234016]: Upload finished. err: <nil> Duration: 6.278258764s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 740.792664ms trace: https://jaeger.dev.storj.io/trace/2d4b8c059832cc79 FAIL (2025-12-16 19:08:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:44796->45.38.24.2:443: read: connection reset by peer Duration: 740.817232ms edge authservice http us-select-1

Orbiter Check History

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)