Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 513975 reports (Page 5419 of 10280)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........]
duration:
6.172527911s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:16:13 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.172572506s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........]
duration:
6.51337531s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:15:04 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.513408502s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........]
duration:
6.208428659s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:14:11 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.208452796s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration:
6.120569196s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:13:06 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.12058675s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration:
30.00567032s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:13:00 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 30.005699085s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration:
5.850805969s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:11:55 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 5.850839372s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration:
5.879832851s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:11:22 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 5.879867516s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...]
duration:
6.279394288s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:10:44 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.279425177s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration:
6.014444093s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:09:46 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.014555755s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration:
6.089652024s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:09:03 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 6.089674188s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X]
duration:
30.00537998s
trace:
https://jaeger.dev.storj.io/trace/c54f29d3c9897cc
FAIL (2025-11-10 19:08:57 UTC)
orbiter warning: operation: DownloadObject
uplink: metaclient: context deadline exceeded
Duration: 30.005403746s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................]
duration:
6.191228216s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:04:46 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.191260538s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration:
7.966638381s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:04:21 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 7.966671203s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration:
6.243308282s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:03:06 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.243336277s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................]
duration:
7.719251211s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:02:11 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 7.719299163s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration:
6.031439433s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:01:23 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.031472667s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
6.352933436s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 19:00:18 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.352982197s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
5.727396026s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:59:57 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.727428948s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration:
6.356273088s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:59:12 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.356375172s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................]
duration:
6.201463054s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:58:57 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.201495376s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
5.94784852s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:58:11 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.947929224s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................]
duration:
6.225172143s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:57:42 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.225205666s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................]
duration:
5.967322612s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:56:32 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.967353781s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................]
duration:
6.285276666s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:55:51 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.285300533s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................]
duration:
6.190490268s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:55:14 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.190579879s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................]
duration:
5.947092609s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:54:20 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.947121514s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................]
duration:
6.051314475s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:53:10 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.051361975s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................]
duration:
5.813746555s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:52:03 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.813776833s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................]
duration:
6.103214126s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:51:15 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.103243722s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration:
5.945962939s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:50:35 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.945982335s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................]
duration:
6.149195648s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:49:22 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.149222911s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration:
6.204635907s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:48:12 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.204666035s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration:
6.202894411s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:47:12 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.202929418s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration:
6.130168946s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:46:45 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.130201448s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration:
6.140233791s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:45:50 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.140266722s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................]
duration:
6.671865534s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:45:13 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.671949795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................]
duration:
6.208619943s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:44:58 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.208652476s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration:
5.990990058s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:44:35 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 5.991019645s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................]
duration:
6.26846876s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:44:28 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.26849953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration:
6.120323514s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:43:55 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.120351106s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................]
duration:
6.496435497s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:43:30 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.496496904s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................]
duration:
6.492770749s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:42:55 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.492802028s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................]
duration:
6.721262148s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:42:23 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.721295221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................]
duration:
6.055131206s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:41:57 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.055166263s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration:
6.499316879s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:41:34 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.49934974s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
6.111514373s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:40:59 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.111545773s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................]
duration:
6.800188171s
trace:
https://jaeger.dev.storj.io/trace/5333a866765d6ba7
FAIL (2025-11-10 18:40:44 UTC)
orbiter warning: operation: ListObjects
uplink: metainfo: metaclient: context deadline exceeded
Duration: 6.800231464s core uplink us1
checkers.HTTP{"GET" "https://www.storj.io/dcs-satellites", ExpectedStatus: 301, ExpectedRedirect: "https://static.storj.io/dcs-satellites"} failure (antiblip window 6/15): (http.www_dcs_satellites_redirect)
antiblip:
1 out of 15 is failed (threshold=6) [X..............]
duration:
297.932539ms
remote:
"tcp:[2620:cb:2000::1]:443"
target:
https://www.storj.io/dcs-satellites
trace:
https://jaeger.dev.storj.io/trace/39f9e251068d3e03
FAIL (2025-11-10 18:40:42 UTC)
orbiter warning: Get "https://www.storj.io/dcs-satellites": read tcp [2400:6180:0:d0::f70:8001]:36580->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 298.002001ms core website
checkers.HTTP{"GET" "https://www.storj.io/"} failure (antiblip window 6/15): (http.www_storj_io)
antiblip:
1 out of 15 is failed (threshold=6) [X..............]
duration:
53.85886ms
remote:
"tcp:[2620:cb:2000::1]:443"
target:
https://www.storj.io/
trace:
https://jaeger.dev.storj.io/trace/6ff6c65486d4bfe7
FAIL (2025-11-10 18:40:41 UTC)
orbiter warning: Get "https://www.storj.io/": read tcp [2400:6180:0:d0::f70:8001]:55052->[2620:cb:2000::1]:443: read: connection reset by peer
Duration: 53.956215ms core website
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
6.170634592s
trace:
https://jaeger.dev.storj.io/trace/43dc1ef093bac9d6
FAIL (2025-11-10 18:39:57 UTC)
orbiter warning: 
operation: Commit
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
false
Nov 10 17:58:50.195246561 (33h54m48.746717022s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 10 17:58:50.195253364 (33h54m48.746723825s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 10 17:58:50.446150738 (33h54m48.997621199s) [seg_pos={0 0}, upload=29890]: Begin upload segment...
Nov 10 17:58:50.446204921 (33h54m48.997675382s) [seg_pos={0 0}, upload=29890]: Done begin upload segment.
Nov 10 17:58:50.446303128 (33h54m48.997773619s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.446324790 (33h54m48.997795251s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447279928 (33h54m48.998750389s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.447694748 (33h54m48.999165199s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448024116 (33h54m48.999494567s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448411413 (33h54m48.999881874s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.448773733 (33h54m49.000244184s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449192981 (33h54m49.000663442s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449208451 (33h54m49.000678912s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449637497 (33h54m49.001107959s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.449875771 (33h54m49.001346232s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450216110 (33h54m49.001686571s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450394219 (33h54m49.00186468s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.450853423 (33h54m49.002323874s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.454582924 (33h54m49.006053385s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:50.460585405 (33h54m49.012055856s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Uploading piece...
Nov 10 17:58:52.149625484 (33h54m50.701095945s) [upload=29890]: Next returned nil segment
Nov 10 17:58:52.149643759 (33h54m50.70111423s) [upload=29890]: Waiting for error group managing segments...
Nov 10 17:58:52.555705621 (33h54m51.107176082s) [noise=true , address=77.90.31.18:4805     , piece_id=432T4MFOFSI4XLSBAWTS4KBHDUUAANMAVGBVLNX6LCN52WV2G2UA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.556717748 (33h54m51.108188219s) [noise=true , address=45.91.116.7:4009     , piece_id=76LEMQOIGP4DGXVJO7BR6BKMBQW4NT7CJVE4KIGUXU6P6HSPJO5A, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.557576213 (33h54m51.109046674s) [noise=true , address=77.90.31.17:4772     , piece_id=NUDHKCIIX2EMFTPMIS4Q7K6VOSRT3O7DQGKE6XEHASSMF335W5KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.576587635 (33h54m51.128058096s) [noise=true , address=45.91.116.6:4974     , piece_id=2MCPOHWTLXOHXRFXNF3UEWIFQY7XD3UUMVGE4BH356LP6DI2663Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.588050924 (33h54m51.139521385s) [noise=true , address=77.90.31.13:4626     , piece_id=IHYVRGDSJMNIQSQIFRFKFMQ4UISCEK7ALER4K3UIW24Y4WBN2DEA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.592877585 (33h54m51.144348046s) [noise=true , address=45.91.116.8:4045     , piece_id=7AOWQRMD3QXQNGTMYMA4IDYBWNX4ZIRMRXSEZYJ4XRZM6YVIQ5NQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:52.623340614 (33h54m51.174811075s) [noise=true , address=77.90.31.14:4663     , piece_id=K5SDMWGZ5LYQ367MDEPFNKCOWMOEPU6AGOGRPZGZSDI27BK4UQ4Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:58:57.489476855 (33h54m56.040947306s) [noise=true , address=145.79.227.9:4367    , piece_id=DB7IT7IS7VU3EFMXUHLXZPGQK5ZD2GLD3KCOOUCKSHGYQVA7AEKA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:06.248396761 (33h55m4.799867222s) [noise=true , address=145.79.227.26:4655   , piece_id=TP4GUPIWUC7XXS6FPQD4AIWHDFKBV4CSUIMTP5RLNF5I6MQD3I6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:09.685941293 (33h55m8.237411754s) [noise=true , address=145.79.227.32:4835   , piece_id=RFABYRUGXNCCO4GZUCPCJBESQR74YV4MXB3PHJH2MQQC7WU3WHQA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.118007415 (33h55m9.669477877s) [noise=true , address=145.79.227.10:4401   , piece_id=X5LBIMPE4K7XWYHWBWU37IN7VSTBXQVSNQXMYBE7H367ZBZCKKVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:11.796179026 (33h55m10.347649487s) [noise=true , address=145.79.227.5:4221    , piece_id=2GI5X6YJ33EKU7XISGS4DLJ2X34LIVNE6NCZMIF7UIUOTGUHYJNQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451686523 (33h55m18.003156984s) [noise=true , address=145.79.227.2:4115    , piece_id=ODSJSZXA5XW5PSARFHFK5IJAJN4X4CLKHI4JEHVMYCBEPP2AG3KA, seg_pos={0 0}, upload=29890]: Done uploading piece. err: <nil>
Nov 10 17:59:19.451715117 (33h55m18.003185588s) [seg_pos={0 0}, upload=29890]: Segment reached optimal threshold of 13 pieces.
Nov 10 17:59:19.451954794 (33h55m18.003425255s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo): protocol: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56452->145.79.227.17:4971: use of closed network connection
Nov 10 17:59:19.451964883 (33h55m18.003435344s) [noise=true , address=145.79.227.17:4971   , piece_id=4Q77MAZ3LE7S5JKQW72H5P7YQL4SLHFKEMP3XC4VXJYYVTBA3IVQ, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452454675 (33h55m18.003925136s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP): protocol: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55722->145.79.227.15:4876: use of closed network connection
Nov 10 17:59:19.452468462 (33h55m18.003938923s) [noise=true , address=145.79.227.15:4876   , piece_id=MW2FFRZE3NS47F6J7BH7J2SFROV7YT4T3WC3D2ECRYO7GY655E6Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452617676 (33h55m18.004088137s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg): protocol: expected piece hash; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; write tcp 188.166.225.170:43630->145.79.227.30:4763: use of closed network connection; EOF
Nov 10 17:59:19.452627334 (33h55m18.004097795s) [noise=true , address=145.79.227.30:4763   , piece_id=HAZUMU2MFCWMN22NA3Z46YBLHRGD3V2N4VQYLTUYEPKL5YGSBL7Q, seg_pos={0 0}, upload=29890]: Long tail cancellation due to optimal threshold
Nov 10 17:59:19.452651190 (33h55m18.004121651s) [seg_pos={0 0}, upload=29890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Nov 10 17:59:19.452756731 (33h55m18.004227192s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Nov 10 17:59:19.452764075 (33h55m18.004234536s) [upload=29890]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Nov 10 17:59:19.740258739 (33h55m18.29172921s) [upload=29890]: Done uploading segments...
Nov 10 17:59:19.740379268 (33h55m18.291849729s) [upload=29890]: Upload finished. err: metaclient: context deadline exceeded
Duration: 6.170661843s core uplink us-select-1