Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
6.252683607s
FAIL (2025-10-06 09:52:17 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.252718394s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.075683154s
FAIL (2025-10-06 09:51:30 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.07571252s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
6.286591991s
FAIL (2025-10-06 09:50:50 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.286643598s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.202506114s
FAIL (2025-10-06 09:50:27 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.202542844s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
6.145521187s
FAIL (2025-10-06 09:49:14 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.145551133s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.053133465s
FAIL (2025-10-06 09:48:29 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.053212044s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
6.201359825s
FAIL (2025-10-06 09:47:39 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.201391224s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
6.164577053s
FAIL (2025-10-06 09:47:19 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.16461169s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.239854223s
FAIL (2025-10-06 09:46:31 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.239892145s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.257821797s
FAIL (2025-10-06 09:45:52 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.257854199s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.565926826s
FAIL (2025-10-06 09:44:51 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.56596017s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.031991735s
FAIL (2025-10-06 09:44:33 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.032022764s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
6.054963146s
FAIL (2025-10-06 09:44:09 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.054998545s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
11.219620899s
FAIL (2025-10-06 09:43:10 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 11.219664322s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.495918671s
FAIL (2025-10-06 09:42:42 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.495941906s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.074650696s
FAIL (2025-10-06 09:42:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 6.074682679s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.000983492s
FAIL (2025-10-06 09:41:28 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Oct 6 09:40:58.992804678 (308h9m44.454897161s) [upload=271187]: Starting upload Oct 6 09:40:58.992908726 (308h9m44.455001209s) [upload=271187]: Done starting upload Oct 6 09:40:58.993221391 (308h9m44.455313874s) [upload=271187]: Uploading segments... Oct 6 09:40:58.993344355 (308h9m44.455436868s) [upload=271187]: Got next segment. Inline: false Oct 6 09:40:58.993429448 (308h9m44.455521931s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 6 09:40:58.993436090 (308h9m44.455528573s) [upload=271187]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Oct 6 09:41:28.298141381 (308h10m13.760233874s) [upload=271187]: Next segment err: context deadline exceeded Oct 6 09:41:28.298287128 (308h10m13.760379611s) [upload=271187]: Done uploading segments... Oct 6 09:41:28.298307447 (308h10m13.76039993s) [upload=271187]: Upload finished. err: context deadline exceeded
Duration: 30.001009461s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.873225627s
FAIL (2025-10-06 05:46:21 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.873263941s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.881602468s
FAIL (2025-10-06 05:45:47 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.88162962s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.851612624s
FAIL (2025-10-06 05:45:25 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.851648382s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.894945938s
FAIL (2025-10-06 05:44:35 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.894983188s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.977355149s
FAIL (2025-10-06 05:43:54 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.977410114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.998977855s
FAIL (2025-10-06 05:43:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 5.999006409s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [XX..........................................................] duration:
6.049599332s
FAIL (2025-10-06 05:42:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.049633957s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.976992272s
FAIL (2025-10-06 05:42:43 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.977023821s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.823695813s
FAIL (2025-10-06 05:42:16 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.823726701s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.XX.........................................................] duration:
5.947863899s
FAIL (2025-10-06 05:42:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 5.94791222s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.727063451s
FAIL (2025-10-06 05:41:49 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.727097545s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..XX........................................................] duration:
6.144522345s
FAIL (2025-10-06 05:40:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.14461357s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.944880255s
FAIL (2025-10-06 05:40:56 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.944933395s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...XX.......................................................] duration:
6.352332254s
FAIL (2025-10-06 05:40:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.352371237s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.029804517s
FAIL (2025-10-06 05:40:03 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 6.029838752s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.963037492s
FAIL (2025-10-06 05:39:13 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.963058501s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [....XX......................................................] duration:
5.961702657s
FAIL (2025-10-06 05:39:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 5.961737213s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration:
6.185765724s
FAIL (2025-10-06 05:38:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.185793337s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
5.936138417s
FAIL (2025-10-06 05:38:06 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.936174597s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
5.765305886s
FAIL (2025-10-06 05:37:15 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.765335523s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......XX....................................................] duration:
6.637249802s
FAIL (2025-10-06 05:37:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.637285758s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......XX...................................................] duration:
6.15961837s
FAIL (2025-10-06 05:36:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.159646675s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
5.826289017s
FAIL (2025-10-06 05:36:05 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.826417521s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........XX..................................................] duration:
6.191676479s
FAIL (2025-10-06 05:35:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.191709532s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.224721336s
FAIL (2025-10-06 05:35:06 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 6.224778936s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........XX.................................................] duration:
6.129731656s
FAIL (2025-10-06 05:34:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.129761714s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........XX................................................] duration:
6.101398137s
FAIL (2025-10-06 05:34:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.101434626s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.903614071s
FAIL (2025-10-06 05:34:13 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 6.903721696s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [...........XX...............................................] duration:
6.091900877s
FAIL (2025-10-06 05:34:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.091932256s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [............XX..............................................] duration:
6.358458258s
FAIL (2025-10-06 05:33:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.358491942s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
5.673243898s
FAIL (2025-10-06 05:32:58 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.673277431s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.............XX.............................................] duration:
6.469861428s
FAIL (2025-10-06 05:32:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
ts...
Oct 6 04:57:41.224043172 (303h26m26.686135655s) [upload=267048]: Got next segment. Inline: false
Oct 6 04:57:41.224102514 (303h26m26.686195007s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Oct 6 04:57:41.224109657 (303h26m26.686202141s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Oct 6 04:57:43.571810484 (303h26m29.033902957s) [seg_pos={0 0}, upload=267048]: Begin upload segment...
Oct 6 04:57:43.571868615 (303h26m29.033961088s) [seg_pos={0 0}, upload=267048]: Done begin upload segment.
Oct 6 04:57:43.571895747 (303h26m29.03398822s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.571905405 (303h26m29.033997878s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572493255 (303h26m29.034585738s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572926419 (303h26m29.035018902s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.572961656 (303h26m29.035054139s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573291805 (303h26m29.035384288s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573314108 (303h26m29.035406581s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573649005 (303h26m29.035741478s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573658683 (303h26m29.035751167s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573968373 (303h26m29.036060856s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.573993301 (303h26m29.036085774s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574246613 (303h26m29.036339096s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574357936 (303h26m29.036450419s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574510025 (303h26m29.036602508s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.574846366 (303h26m29.036938839s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:43.575267768 (303h26m29.037360261s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Uploading piece...
Oct 6 04:57:45.223767638 (303h26m30.685860122s) [upload=267048]: Next returned nil segment
Oct 6 04:57:45.223788298 (303h26m30.685880781s) [upload=267048]: Waiting for error group managing segments...
Oct 6 04:57:45.610930742 (303h26m31.073023225s) [noise=true , address=145.79.227.13:4804 , piece_id=ZY22V7MPCZAITPCOILYZMFD3M3PJKOS3IQZCGLWEAP56OE4OTP6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.616752047 (303h26m31.078844521s) [noise=true , address=145.79.227.14:4837 , piece_id=PAK47UNOBAZEK5BO6AI2PYRD5FNTM5YMC6GHCFAAFMNKVYESRYHQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.632692851 (303h26m31.094785334s) [noise=true , address=145.79.227.16:4912 , piece_id=FL5OGZXETDNLNIPZZVX7FJP4XJEDPOZGLZPTGAK62AVFUZOIHKMA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.741911141 (303h26m31.204003614s) [noise=true , address=45.91.116.4:4939 , piece_id=VSSYYGEJHOM7ZBGQLEIQFADDTTEKTA2BAZVMTKPL3BDERBBB53LA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.767861796 (303h26m31.229954269s) [noise=true , address=45.91.116.12:4226 , piece_id=7ZK2XTMQRGMHQ5OTVL7CC6KF3OV5AJGGCZ2ALHEKPGEOCWHFY2UQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.780662230 (303h26m31.242754713s) [noise=true , address=45.91.116.6:4973 , piece_id=T42FY7UFEUUUF7L4RVNOEUSAJWMAHWWCDRNYDIF7ITJMYTRDSVGA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.781386108 (303h26m31.243478591s) [noise=true , address=77.90.31.18:4807 , piece_id=OFCM262ILPSJYXK2WWQDTHLMN6ZEIOXTPE2IIZO5TY5RW46MUALQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.794405830 (303h26m31.256498313s) [noise=true , address=45.91.116.7:4009 , piece_id=LWZ2DVOM5SGFS42265C76ZR526KBJ5VUDUBJWH64CZVXAS6OUPPA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.812311745 (303h26m31.274404228s) [noise=true , address=77.90.31.11:4555 , piece_id=D7NTZTR2D32X7D2RTHPQLMMCWKYV7SH67QUSNJCWPLWTCBSJQB6A, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.831180014 (303h26m31.293272497s) [noise=true , address=77.90.31.15:4699 , piece_id=SM43TCBHLWAWMWVRJSMXRTMJS33OJITTEINPNHGU4QGXMORMWZPQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.866429014 (303h26m31.328521497s) [noise=true , address=45.91.116.14:4191 , piece_id=ZRAW6JVVCJZBPIZ2NDPYVOA7NFOPOYNNRSOO5A6SRHOEC7ZWCPXQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:45.952301860 (303h26m31.414394343s) [noise=true , address=77.90.31.21:4877 , piece_id=FRJUNNVDMBNSSNJETRCXMMHLNZBZFPTFGSXU7QC5LBJEMFGAG6JQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142182876 (303h26m31.604275359s) [noise=true , address=45.91.116.6:4975 , piece_id=YZRQHR3U56UNBEHYZDIBRI6RT3HWTCDJYQNBU47XJNSG2ZRVOKDQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: <nil>
Oct 6 04:57:46.142218714 (303h26m31.604311197s) [seg_pos={0 0}, upload=267048]: Segment reached optimal threshold of 13 pieces.
Oct 6 04:57:46.142376203 (303h26m31.604468686s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:46.142389229 (303h26m31.604481712s) [noise=true , address=45.91.116.9:4263 , piece_id=4M43WAB7WSXA76HHNNKQ7NLI4ONQUZD5JEP2TOCIRJ5EI7CN3QRQ, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142503406 (303h26m31.604595919s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56482->45.91.116.13:4155: use of closed network connection
Oct 6 04:57:46.142516290 (303h26m31.604608773s) [noise=true , address=45.91.116.13:4155 , piece_id=XOYNKHBX3QYXMDZGJXFHRIFLUWP6A4DJZT4GUKTORQNLW6KSRFAA, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142544694 (303h26m31.604637177s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2): protocol: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49750->77.90.31.17:4771: use of closed network connection
Oct 6 04:57:46.142554794 (303h26m31.604647277s) [noise=true , address=77.90.31.17:4771 , piece_id=APNOQTEN7V27O47TBFUI3LT74XMRYYL6NCE5EZOJLK2SIHPMK44Q, seg_pos={0 0}, upload=267048]: Long tail cancellation due to optimal threshold
Oct 6 04:57:46.142578448 (303h26m31.604670922s) [seg_pos={0 0}, upload=267048]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Oct 6 04:57:46.142674181 (303h26m31.604766664s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:46.142685262 (303h26m31.604777745s) [upload=267048]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:46.496730243 (303h26m31.958822726s) [upload=267048]: Done uploading segments...
Oct 6 04:57:46.496750101 (303h26m31.958842584s) [upload=267048]: Upload finished. err: <nil>
Duration: 6.46989389s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
5.770444169s
FAIL (2025-10-06 05:32:09 UTC)
orbiter warning:
operation: DownloadObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
261s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.525894000 (303h26m14.987986483s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526161108 (303h26m14.988253581s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526431944 (303h26m14.988524417s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.526728489 (303h26m14.988820962s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527001840 (303h26m14.989094323s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527272805 (303h26m14.989365278s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527690110 (303h26m14.989782593s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.527989380 (303h26m14.990081863s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528434017 (303h26m14.9905265s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.528840129 (303h26m14.990932602s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.529969140 (303h26m14.992061613s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.531177089 (303h26m14.993269562s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.532437410 (303h26m14.994529883s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.536309833 (303h26m14.998402317s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:29.539614767 (303h26m15.00170725s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Uploading piece...
Oct 6 04:57:30.894336315 (303h26m16.356428798s) [upload=267045]: Next returned nil segment
Oct 6 04:57:30.894358266 (303h26m16.356450739s) [upload=267045]: Waiting for error group managing segments...
Oct 6 04:57:31.242860152 (303h26m16.704952625s) [noise=true , address=64.49.13.139:4469 , piece_id=X6INPYLO2OXKQAZXMTEDSRP7WHO6YQAWF3RVZODRUY26TOSVWN5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.459373200 (303h26m16.921465673s) [noise=true , address=145.79.227.36:4542 , piece_id=7GRYK7MFO5THQNBVM5IR4HA2QK3B5PKSDFNJ4DI5N36OMLIULLOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.462680437 (303h26m16.92477292s) [noise=true , address=145.79.227.24:4600 , piece_id=ZIKYHYXFUT77L4NDVO7JBWYR34PN6OCOV7J5YVHZ35MZZMOVTTFA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.477541926 (303h26m16.939634399s) [noise=true , address=145.79.227.30:4796 , piece_id=7G37GRMHYOOVYYWHYTUSEZ55NKDMJYDO7ZDAY54FFBVMA5I7LS5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.492660103 (303h26m16.954752577s) [noise=true , address=145.79.227.39:4029 , piece_id=QYRKK4DYWI5IPDU4VWNTMF6QIXQOX3ZXZFFO7M5ZC5IMBZOHTYXA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.527656232 (303h26m16.989748735s) [noise=true , address=77.90.31.15:4707 , piece_id=46SNORHYVSUK3O5EW6CAGK3BPFNXM4RIHSLOF5A6Q47JXFSIWMEA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.537601379 (303h26m16.999693862s) [noise=true , address=77.90.31.19:4867 , piece_id=R7WVGDCYCAM4CZT6ZCA4FSGDANGDZ4KYWK5FI4WP6JRDPXSHMEIA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.565592068 (303h26m17.027684552s) [noise=true , address=45.91.116.4:4963 , piece_id=IKEGQMYR3JHVKR6DGKJBKCBT5O55HVNHLDZLQHANLQ3Y6X64EZOQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.571773218 (303h26m17.033865701s) [noise=true , address=77.90.31.14:4667 , piece_id=EYJLH6IN6QKY6PXE6I7JHT7DP7RN5PITP7NK6CXV4GRB2SZ44Y4Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.572376567 (303h26m17.03446905s) [noise=true , address=45.91.116.5:4316 , piece_id=UH7DHBJ5RNU4HLWP4WKJGGMZFT4VEX4F5HFDVQOX5QA5NV6JZX5Q, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.578533350 (303h26m17.040625833s) [noise=true , address=45.91.116.11:4125 , piece_id=Q3LWE6OUXBZLAMGILMK3NEE56RICRS7DBAVDPHEAFDW2YLUXSB3A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.610757401 (303h26m17.072849894s) [noise=true , address=45.91.116.5:4326 , piece_id=VR7AM335ZLN2SXZWGBQCMIDVSEGDUJTBLGLIVUTASLNLZQ53J7EA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.621005394 (303h26m17.083097877s) [noise=true , address=77.90.31.13:4645 , piece_id=Y3IWUZFQZ2CVXSCJQOMCIEIV4REMOY2WLKPA2ZH77KQFTJC3NOKQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639189480 (303h26m17.101281963s) [noise=true , address=77.90.31.17:4776 , piece_id=OVALY7AMN5UU3DVXVK6O7Y5VXQ3A7JS6TRUMRZZMILI4PJQPC6VA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: <nil>
Oct 6 04:57:31.639217743 (303h26m17.101310216s) [seg_pos={0 0}, upload=267045]: Segment reached optimal threshold of 14 pieces.
Oct 6 04:57:31.639354163 (303h26m17.101446636s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639368982 (303h26m17.101461455s) [noise=true , address=45.91.116.14:4217 , piece_id=BQVE7APQNKBDLACD4G5E75P5O7RD6H6EEHAHBF2I2EUFIXUYWNAQ, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639509490 (303h26m17.101601993s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12BQKHUMVYD7qDZW45LNEfgnkLVoSneDrQXViaMvcvAPdfmheMq): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639542122 (303h26m17.101634595s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1P53pBJhZqd4Kn47H6UcmxkDSh11qKmUaMhKTk3bGBF4FbmgMC): protocol: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41500->77.90.31.15:4718: use of closed network connection
Oct 6 04:57:31.639591315 (303h26m17.101683798s) [noise=true , address=45.91.116.7:4027 , piece_id=SY37LF5EAHAS6CU4RZJ2MIWKULX3NEFGR7TRUX5AULH7PL4VI2XA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639616864 (303h26m17.101709357s) [noise=true , address=77.90.31.15:4718 , piece_id=RLSHR5I7T546M5NIAV4EQRTIMPLPYU54SCQBD32LNQ5PA7RECZAA, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639672359 (303h26m17.101764842s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA): protocol: expected piece hash; context canceled; EOF
Oct 6 04:57:31.639682990 (303h26m17.101775483s) [noise=true , address=77.90.31.10:4540 , piece_id=N2ZSAGV6O2HEUPHO54A7GGDPAGILRJYYGFE5RQSY6XS7L4IH323A, seg_pos={0 0}, upload=267045]: Long tail cancellation due to optimal threshold
Oct 6 04:57:31.639739357 (303h26m17.10183184s) [seg_pos={0 0}, upload=267045]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Oct 6 04:57:31.639863173 (303h26m17.101955676s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Oct 6 04:57:31.639870687 (303h26m17.10196317s) [upload=267045]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Oct 6 04:57:33.504069625 (303h26m18.966162108s) [upload=267045]: Done uploading segments...
Oct 6 04:57:33.504089093 (303h26m18.966181566s) [upload=267045]: Upload finished. err: <nil>
Duration: 5.770474355s
core
uplink
us-select-1
us1-placement-20
|