Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
7.660907465s
FAIL (2025-10-20 14:40:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.660948143s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
7.077662617s
FAIL (2025-10-20 14:39:27 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.077697312s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
2.157762427s
FAIL (2025-10-20 14:39:08 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.157792194s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
6.792797235s
FAIL (2025-10-20 14:38:57 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.792832042s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.670050041s
FAIL (2025-10-20 14:38:13 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.67009167s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
2.092203799s
FAIL (2025-10-20 14:37:54 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.092231793s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.368473763s
FAIL (2025-10-20 14:37:53 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.368512686s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.689768631s
FAIL (2025-10-20 14:37:15 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.689805971s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
2.126680465s
FAIL (2025-10-20 14:36:44 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.126713989s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.585391799s
FAIL (2025-10-20 14:36:18 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.585498652s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
2.128214035s
FAIL (2025-10-20 14:36:00 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.128248641s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.417285466s
FAIL (2025-10-20 14:35:40 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.417321586s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
2.114959475s
FAIL (2025-10-20 14:35:16 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.114995134s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
7.622818491s
FAIL (2025-10-20 14:35:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.622857265s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
2.183981336s
FAIL (2025-10-20 14:35:00 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.184082578s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
6.816710492s
FAIL (2025-10-20 14:33:58 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.816748272s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
2.190207882s
FAIL (2025-10-20 14:33:55 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.190242749s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
2.117235089s
FAIL (2025-10-20 14:33:29 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.117268081s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.715854703s
FAIL (2025-10-20 14:33:13 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.715890973s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
2.085714196s
FAIL (2025-10-20 14:32:46 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.085747108s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
7.02713563s
FAIL (2025-10-20 14:32:33 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.027168793s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
2.178102822s
FAIL (2025-10-20 14:32:31 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.178171744s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
2.215442513s
FAIL (2025-10-20 14:31:43 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.215479804s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
6.716894275s
FAIL (2025-10-20 14:31:30 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.716935625s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
4.972914112s
FAIL (2025-10-20 14:31:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.972949069s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
2.071649427s
FAIL (2025-10-20 14:30:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.071712015s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.468037273s
FAIL (2025-10-20 14:30:38 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.468073722s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
2.184820513s
FAIL (2025-10-20 14:30:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.184844469s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
4.979578994s
FAIL (2025-10-20 14:30:23 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.979606576s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
2 out of 60 is failed (threshold=3) [X.X.........................................................] duration:
5.420353062s
FAIL (2025-10-20 14:30:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 5.420392967s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
7.771929808s
FAIL (2025-10-20 14:30:06 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.771969826s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
3 out of 60 is failed (threshold=3) [XX.X........................................................] duration:
4.988226279s
FAIL (2025-10-20 14:29:29 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.988262898s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
2.229940702s
FAIL (2025-10-20 14:29:26 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.229972623s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.985734226s
FAIL (2025-10-20 14:29:26 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.985766156s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
4 out of 60 is failed (threshold=3) [XXX.X.......................................................] duration:
4.996432788s
FAIL (2025-10-20 14:29:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.996556683s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
5 out of 60 is failed (threshold=3) [XXXX.X......................................................] duration:
5.004806592s
FAIL (2025-10-20 14:28:49 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 5.004845476s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
2.102438334s
FAIL (2025-10-20 14:28:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.102479552s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
7.098845461s
FAIL (2025-10-20 14:28:26 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.098879775s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [XXXXX.X.....................................................] duration:
5.000944837s
FAIL (2025-10-20 14:28:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 5.000977288s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.775521288s
FAIL (2025-10-20 14:28:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.775557697s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [.XXXXX.X....................................................] duration:
5.084143183s
FAIL (2025-10-20 14:28:04 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 5.084177668s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
6.7482929s
FAIL (2025-10-20 14:27:32 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 6.74832961s
core
uplink
us1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
2.11887894s
FAIL (2025-10-20 14:27:23 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.118914898s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [..XXXXX.X...................................................] duration:
4.987192679s
FAIL (2025-10-20 14:27:15 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.987228578s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
7.413981286s
FAIL (2025-10-20 14:27:04 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.41401462s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [...XXXXX.X..................................................] duration:
4.88453975s
FAIL (2025-10-20 14:26:58 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 4.884574698s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
2.097474061s
FAIL (2025-10-20 14:26:31 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.09751522s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
7.307819154s
FAIL (2025-10-20 14:26:10 UTC)
orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded
Duration: 7.307881462s
core
uplink
us1
|
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)antiblip:
6 out of 60 is failed (threshold=3) [....XXXXX.X.................................................] duration:
5.002155612s
FAIL (2025-10-20 14:26:08 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 26 but got 22)
======LOGS======
AAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.844814185 (648h15m55.30690667s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845181405 (648h15m55.307273888s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.845581047 (648h15m55.30767352s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847009346 (648h15m55.309101829s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.847967961 (648h15m55.310060444s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.848679988 (648h15m55.310772461s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.849458780 (648h15m55.311551273s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:09.850358614 (648h15m55.312451097s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Uploading piece...
Oct 20 13:47:10.665425409 (648h15m56.127517892s) [upload=570318]: Next returned nil segment
Oct 20 13:47:10.665773723 (648h15m56.127866236s) [upload=570318]: Waiting for error group managing segments...
Oct 20 13:47:11.059138312 (648h15m56.521230795s) [noise=true , address=81.171.14.70:4044 , piece_id=IU7BHOLOYSUCNYKKMUPUKF2MTQW5U2JSAONP7GT6IRVBJ3FZSZTA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.059772429 (648h15m56.521864912s) [noise=true , address=89.149.201.147:4024 , piece_id=W4JRXHBBQXN6DJR7XHHBMSHAUDC6YCV4EBDJVFP2OSUB5MOVMGEQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.060623740 (648h15m56.522716224s) [noise=true , address=89.149.201.147:4015 , piece_id=3IZA2YR6BZJU772I4XBJ4CFBRKVBJL2W7CGGM64GLKBSIL24767A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065156371 (648h15m56.527248854s) [noise=true , address=81.171.14.70:4025 , piece_id=A4FHZGGZKJIJSTOLK2KZIRNFNGBMC2DR5DGUAV4WVCER2DRHDP5Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.065204332 (648h15m56.527296805s) [noise=true , address=89.149.201.147:4005 , piece_id=EWBJOIZ66JPSNI2AXDTO67EMJ4DGAMGA7JSQSZK47Q2SU4LJL2GA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.075074337 (648h15m56.53716682s) [noise=true , address=81.171.14.70:4029 , piece_id=OFHVJFKRP5KSJHXGSMNIJJVGIP72KF7RAUH4ZCTAPVPAU2Y7J2KA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.080600869 (648h15m56.542693353s) [noise=true , address=81.171.14.70:4048 , piece_id=GYH3LAVLD2QBS2VPKG7MZZHAP5AVFSDZUFILT5DHRL4KTNPDBC4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.088746180 (648h15m56.550838653s) [noise=true , address=81.171.14.70:4036 , piece_id=ZPNEJ2M2KGPYOZOXSOZFPHALN6FOTFFDAAW67HVXIBBGADYZ7SUA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.108467914 (648h15m56.570560397s) [noise=true , address=81.171.14.70:4033 , piece_id=PUFN3Z2IVZEFNUIVT5XBXAZES54Q5XK32SD6KLIK6AD7ZRYF5I3Q, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.117338084 (648h15m56.579430567s) [noise=true , address=89.149.201.147:4006 , piece_id=UT4XCJ6IYTBYBOW6SGSZEGGMRVACPUPINNRQKJLWNJ5VW6PQVFRA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564152612 (648h15m57.026245095s) [noise=true , address=81.171.14.70:4031 , piece_id=QM4CB5HTQSFBSUQGOHR5D5N7DXVBCOMS2NSCGAF7WOKFYUFRHNCQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.564194822 (648h15m57.026287305s) [noise=true , address=81.171.14.70:4037 , piece_id=B3Y67GNW2JDZSJ7V7F3LRJKNJP3A42QFJC4KBXQRC3SDVREY2LPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.574721636 (648h15m57.036814129s) [noise=true , address=89.149.201.147:4008 , piece_id=I43C25LCSLUPGNJ6JRA5U6VF2ABX4ZB4IPIAFQTMLC23S44WLOVA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.600155156 (648h15m57.062247639s) [noise=true , address=89.149.201.147:4013 , piece_id=WCW53MNXPWTDN25FOQV6F6GTLPF6OYJUQMPM33R2PQ3OTKAJ6QZQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.698932968 (648h15m57.161025451s) [noise=true , address=185.167.204.206:4056 , piece_id=ZRUXQ4AUFKASSZ7Z2VYIB4INQUD63KJDVB5KJSNQ73YMBE6VYZJA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.702794421 (648h15m57.164886894s) [noise=true , address=185.167.204.206:4041 , piece_id=QWV4ATZ3ZNE2ZBD4XFSFGEDDMIZAIQY7ZVFJPPB5EQC7YBI6YFRQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.704061294 (648h15m57.166153767s) [noise=true , address=185.167.204.206:4046 , piece_id=OWYIAONWKQDIEPEKMRRFZ2N3XEJTGJRMEDIRQM6FJ66WDD3K2YNQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.714100820 (648h15m57.176193303s) [noise=true , address=185.167.204.206:4050 , piece_id=Z7HSSQBFXFJBZPCKPREWAWDTM4O4MPSXM5E33MWGC4W5Y543DB4A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.729206474 (648h15m57.191298987s) [noise=true , address=185.167.204.206:4048 , piece_id=CUKSQ7LF7CGFRWDZR4R6EJ4SJQJFZOJ3RZ7MNDDCCZSNF5FJ7EXA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.736730451 (648h15m57.198822934s) [noise=true , address=185.167.204.205:4033 , piece_id=6WL27JDJOAO3UO6IGTRG2CB2PHPEVTSGPDRBHKM2KLRQ5NGAJ7QQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.748231512 (648h15m57.210323985s) [noise=true , address=185.167.204.206:4052 , piece_id=3GVARUCOXLJKD7YVGAIYTURONORGKUCL3Q2V475YNY2XKJHVG25A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:47:11.779834700 (648h15m57.241927183s) [noise=true , address=185.167.204.205:4024 , piece_id=2RCRZBLZEMLNNLRP7HRP2ZZZ4HRO7BPD4ZO7LNHANIKZW5DRRPTQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: <nil>
Oct 20 13:48:09.083911606 (648h16m54.546004089s) [noise=true , address=185.167.204.205:4018 , piece_id=W5JZIVVG4X6UX4MQBVLVMDMTFXGDJVBYSSKQ7UXRFVC7XJ4LXVPQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12w2LyfAtY3ihjJgkBtord4fMMfbEvz73nExjTNZhLQXurdA3iS, address:185.167.204.205:4018): protocol: rpc: dial tcp 185.167.204.205:4018: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084057464 (648h16m54.546149937s) [noise=true , address=185.167.204.206:4043 , piece_id=7H5FREQWNAEXXNLYO7QBH55ILBN6U63RE7SNA6WHH5IQSHSGSSFQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1EtbRMgkYPVZWSW6hTfGghxDoetX26Fm1Rw97n49jR1oMYt1kb, address:185.167.204.206:4043): protocol: write tcp 188.166.225.170:47744->185.167.204.206:4043: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084177452 (648h16m54.546269925s) [noise=true , address=185.167.204.205:4035 , piece_id=7CEB5LCBKPO4XAAHRNLGLD6GKTZRNRTAFDA34HJCXXL3UXBDW55A, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1TT3Cyz1GPrGa9P9jWePFarF4LFaJfyfRbGM1hH5ctGx65JUJp, address:185.167.204.205:4035): protocol: rpc: dial tcp 185.167.204.205:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084331987 (648h16m54.54642447s) [noise=true , address=185.167.204.206:4070 , piece_id=G47I433AK7THDFNHXN5AYH7NOAK2SAUY3OTENQUEAR4P7NO2JVWQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1HjpnutE3cSKeeVw3nEvoPPAbJKjP6VZYMDGLeT9zSskkwzBUz, address:185.167.204.206:4070): protocol: rpc: dial tcp 185.167.204.206:4070: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084450162 (648h16m54.546542645s) [noise=true , address=185.167.204.206:4045 , piece_id=DDDBPGDQ4OKM3MTJDT6ZZWGZC6MTEDGKL6CUESHIUAJN3WP5ANVQ, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:1JPbi9CNAvezq91fdgjboVcgrdwwN48dFfLXzzgXtJ9B5eEoYq, address:185.167.204.206:4045): protocol: write tcp 188.166.225.170:42380->185.167.204.206:4045: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084557716 (648h16m54.54665019s) [noise=true , address=185.167.204.205:4006 , piece_id=4RSNFFAWW7C4ZHIJG4CEE5ZWISOXBOAEECMGWETLCO4R7VOMRDEA, seg_pos={0 0}, upload=570318]: Done uploading piece. err: ecclient: upload failed (node:12mQUrzCLU4imW5BNBpkSWnRtUftKav85Xi2RsEBacP31AXJ7R8, address:185.167.204.205:4006): protocol: write tcp 188.166.225.170:56046->185.167.204.205:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Oct 20 13:48:09.084580450 (648h16m54.546672923s) [seg_pos={0 0}, upload=570318]: Done waiting for segment. successful: 22 optimal: 26 errs: <nil>
Oct 20 13:48:09.277031502 (648h16m54.739123995s) [upload=570318]: Done uploading segments...
Oct 20 13:48:09.277064935 (648h16m54.739157418s) [upload=570318]: Upload finished. err: failed to upload enough pieces (needed at least 26 but got 22)
Duration: 5.002193915s
core
uplink
eu-select-1
eu1-placement-15
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
2.469613291s
FAIL (2025-10-20 14:25:23 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Oct 20 14:03:56.569391000 (648h32m42.031483483s) [upload=570561]: Starting upload Oct 20 14:03:56.569485550 (648h32m42.031578033s) [upload=570561]: Done starting upload Oct 20 14:03:56.569564411 (648h32m42.031656894s) [upload=570561]: Uploading segments... Oct 20 14:03:56.569617461 (648h32m42.031709944s) [upload=570561]: Got next segment. Inline: true Oct 20 14:03:56.569623472 (648h32m42.031715955s) [upload=570561]: Waiting for error group managing segments... Oct 20 14:03:56.569679279 (648h32m42.031771762s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Oct 20 14:03:56.569685841 (648h32m42.031778314s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Oct 20 14:03:56.569690159 (648h32m42.031782643s) [upload=570561]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Oct 20 14:04:14.795439690 (648h33m0.257532163s) [upload=570561]: Done uploading segments... Oct 20 14:04:14.795463344 (648h33m0.257555817s) [upload=570561]: Upload finished. err: metaclient: context deadline exceeded
Duration: 2.469650963s
core
uplink
us-select-1
|