Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
6.305584574s
FAIL (2025-09-01 15:00:37 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.305610944s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
6.15669456s
FAIL (2025-09-01 14:59:43 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.156735159s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
6.084174722s
FAIL (2025-09-01 14:58:53 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.084201583s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
6.004860512s
FAIL (2025-09-01 14:58:33 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.004891732s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.92360839s
FAIL (2025-09-01 14:58:18 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 5.923631184s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
6.176165841s
FAIL (2025-09-01 14:57:16 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.176197322s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
6.049655474s
FAIL (2025-09-01 14:56:35 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.049679892s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
6.320373175s
FAIL (2025-09-01 14:55:46 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 6.320391319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001274088s
FAIL (2025-09-01 14:55:39 UTC)
orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== Sep 1 14:54:40.612982766 (72h46m37.293178335s) [upload=63963]: Starting upload Sep 1 14:54:40.613041468 (72h46m37.293237037s) [upload=63963]: Done starting upload Sep 1 14:54:40.613125308 (72h46m37.293320877s) [upload=63963]: Uploading segments... Sep 1 14:54:40.615198454 (72h46m37.295394023s) [upload=63963]: Got next segment. Inline: false Sep 1 14:54:40.615236097 (72h46m37.295431666s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Sep 1 14:54:40.615240676 (72h46m37.295436245s) [upload=63963]: Flush batch item: *pb.BatchRequestItem_SegmentBegin Sep 1 14:55:39.626039762 (72h47m36.306235331s) [upload=63963]: Next segment err: context deadline exceeded Sep 1 14:55:39.626212132 (72h47m36.306407701s) [upload=63963]: Done uploading segments... Sep 1 14:55:39.626246888 (72h47m36.306442457s) [upload=63963]: Upload finished. err: context deadline exceeded
Duration: 1m0.001327952s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.369340974s
FAIL (2025-09-01 14:25:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.369406399s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
6.332424122s
FAIL (2025-09-01 14:24:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.33244906s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
7.084526048s
FAIL (2025-09-01 14:23:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.0845533s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.2446319s
FAIL (2025-09-01 14:22:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.244664944s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.216397189s
FAIL (2025-09-01 14:21:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.216424751s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
7.596299512s
FAIL (2025-09-01 14:21:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.596325552s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.594688954s
FAIL (2025-09-01 14:20:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.594782974s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
6.379467045s
FAIL (2025-09-01 14:19:46 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.379514566s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
7.005305136s
FAIL (2025-09-01 14:18:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.005330404s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.260708913s
FAIL (2025-09-01 14:17:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.260732799s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
7.035858367s
FAIL (2025-09-01 14:16:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.035885279s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
8.495205097s
FAIL (2025-09-01 14:15:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 8.495236508s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
16.991299338s
FAIL (2025-09-01 14:14:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 16.991325327s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
7.148142315s
FAIL (2025-09-01 14:14:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.148167843s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
6.319085191s
FAIL (2025-09-01 14:13:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.319206944s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
6.615007688s
FAIL (2025-09-01 14:12:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.615034439s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
6.627771337s
FAIL (2025-09-01 14:11:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.627801174s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
6.409119812s
FAIL (2025-09-01 14:10:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.409147915s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
6.71679736s
FAIL (2025-09-01 14:10:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.716830183s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
6.520798794s
FAIL (2025-09-01 14:09:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.52081775s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
7.004598579s
FAIL (2025-09-01 14:09:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.004622264s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
6.723879349s
FAIL (2025-09-01 14:08:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.723905951s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
7.044276802s
FAIL (2025-09-01 14:08:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.044298506s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
7.221092731s
FAIL (2025-09-01 14:07:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.221132717s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
6.751053905s
FAIL (2025-09-01 14:06:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.751115352s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
6.33794833s
FAIL (2025-09-01 14:05:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.337976295s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
6.358947515s
FAIL (2025-09-01 14:04:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.358971793s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
6.317635211s
FAIL (2025-09-01 14:03:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.317658535s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
6.69683101s
FAIL (2025-09-01 14:02:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.696858442s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
7.08845286s
FAIL (2025-09-01 14:01:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.088478238s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
7.005180848s
FAIL (2025-09-01 14:01:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.005205245s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
6.329667872s
FAIL (2025-09-01 14:00:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.329694643s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
6.671444222s
FAIL (2025-09-01 13:59:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.671471655s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
6.54573178s
FAIL (2025-09-01 13:58:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.545788037s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
6.485985623s
FAIL (2025-09-01 13:57:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.48601025s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
6.965201072s
FAIL (2025-09-01 13:57:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.965224657s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
6.477616548s
FAIL (2025-09-01 13:56:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.477650623s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
6.83210563s
FAIL (2025-09-01 13:55:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.832133915s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
6.802577289s
FAIL (2025-09-01 13:55:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 6.802608208s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
7.136371864s
FAIL (2025-09-01 13:54:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.136399867s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
7.328886812s
FAIL (2025-09-01 13:54:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
Sep 1 13:39:03.243629081 (71h30m59.92382466s) [upload=62844]: Starting upload
Sep 1 13:39:03.243723442 (71h30m59.923919021s) [upload=62844]: Done starting upload
Sep 1 13:39:03.243894589 (71h30m59.924090189s) [upload=62844]: Uploading segments...
Sep 1 13:39:03.244049386 (71h30m59.924244955s) [upload=62844]: Got next segment. Inline: false
Sep 1 13:39:03.244093780 (71h30m59.924289369s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Sep 1 13:39:03.244099993 (71h30m59.924295572s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Sep 1 13:39:03.493449717 (71h31m0.173645286s) [seg_pos={0 0}, upload=62844]: Begin upload segment...
Sep 1 13:39:03.493497088 (71h31m0.173692667s) [seg_pos={0 0}, upload=62844]: Done begin upload segment.
Sep 1 13:39:03.493521354 (71h31m0.173716923s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493567753 (71h31m0.173763322s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.493931799 (71h31m0.174127368s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494060415 (71h31m0.174255985s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494191787 (71h31m0.174387356s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494441014 (71h31m0.174636583s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494558428 (71h31m0.174753997s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494682907 (71h31m0.174878486s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494914590 (71h31m0.175110159s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.494953735 (71h31m0.175149304s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495180268 (71h31m0.175375837s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495189225 (71h31m0.175384794s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495414547 (71h31m0.175610116s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495463600 (71h31m0.175659169s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495650718 (71h31m0.175846287s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:03.495736823 (71h31m0.175932392s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Uploading piece...
Sep 1 13:39:05.329889839 (71h31m2.010085428s) [upload=62844]: Next returned nil segment
Sep 1 13:39:05.329914555 (71h31m2.010110124s) [upload=62844]: Waiting for error group managing segments...
Sep 1 13:39:05.639934400 (71h31m2.320129969s) [noise=true , address=104.36.82.8:4970 , piece_id=IVNGJWXXSNQEJTCPSJZPWFRODZEQYHFC5ZVXGYP7MYCBRDQCXGAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.727090147 (71h31m2.407285716s) [noise=true , address=104.36.82.3:4767 , piece_id=YMHKWJ4AMROCP73BIBDOKGM2NNGYMUA5USS3TFH4U3MTMNOQ6DMA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.728165605 (71h31m2.408361174s) [noise=true , address=104.36.82.15:4257 , piece_id=C4SIPIID6HBZHEQLAFI3ON5BLRSCD53EC66UF3NXKLMNPUZDDUMQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:05.831217087 (71h31m2.511412656s) [noise=true , address=45.91.116.10:4082 , piece_id=QGNOACVAD5QXO277CGY67NEEUI2YE6EXPL6S6EUA5DBSEBU34HAA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.033520572 (71h31m2.713716141s) [noise=true , address=45.91.116.4:4937 , piece_id=QMXSL5GVYOX5UG75N5JMMILBUDA7KZJJWVLL7GHCM3QZXQG4PCXQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.049153966 (71h31m2.729349535s) [noise=true , address=45.91.116.6:4974 , piece_id=7LUPZLAD2AKPVAGC3IAHCXPBNXNH3N33SN7JURZWYTL33M7OQPWA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.082056062 (71h31m2.762251631s) [noise=true , address=45.91.116.13:4154 , piece_id=AIMZVQVXNPFJNBEDUPT7WFBO52M2HSOQDO7DKTJQ4RRTWQRV6JYQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.162607342 (71h31m2.842802911s) [noise=true , address=45.91.116.5:4299 , piece_id=DKXDLK54TLCD3GT6GT2HO7VFDF6KBBRWMMOXH3MKVHSHTSPUY3BQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.278978791 (71h31m2.95917436s) [noise=true , address=45.91.116.11:4117 , piece_id=IF3XWHKWOWKOW7K6H7QXIZ5YGNM2QOZTY7ZDCIVZDWZYVVW2CA3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.479588907 (71h31m3.159784476s) [noise=true , address=45.91.116.12:4227 , piece_id=A4IA6A2S7HBH5YCAGHMIFHZMA2VMTLSRIACFKVSU4E62WOKLDHWQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.810348935 (71h31m3.490544555s) [noise=true , address=77.90.31.17:4770 , piece_id=O4W767BHA6LX53F3LLPJFYAWVORERKFYIZV6ETOJM5EHYC6H5NRQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.882428885 (71h31m3.562624455s) [noise=true , address=45.91.116.4:4939 , piece_id=MQRMMGQVZNMCNMEQ6VPHADCUOWSQNT3XVVRXIUKYNEJEKL6RBX6A, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891868357 (71h31m3.572063916s) [noise=true , address=77.90.31.15:4698 , piece_id=R4YAZZMYHUAJMBKOV57XQZCDWYEUCDQY3VVQMHHX53KAL6A23F3Q, seg_pos={0 0}, upload=62844]: Done uploading piece. err: <nil>
Sep 1 13:39:06.891896430 (71h31m3.572091999s) [seg_pos={0 0}, upload=62844]: Segment reached optimal threshold of 13 pieces.
Sep 1 13:39:06.892004808 (71h31m3.572200377s) [noise=true , address=77.90.31.19:4841 , piece_id=OOTCE7RQNPBD7UGHB62G2ZP63ZS72CE7GYT3W3VJRSIEXOOHUHDQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892035097 (71h31m3.572230666s) [noise=true , address=77.90.31.15:4699 , piece_id=LZI3FWV74BNS2IUKD23NVQF6XTQOAXSHD6CWA6R3B3RX7AJDBRTA, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892056938 (71h31m3.572252517s) [noise=true , address=104.36.82.13:4186 , piece_id=6EYOHHWMHFVSKVUXJVMXE7LYPM6E64YRT6F5BPH2BBOPTC7BKWLQ, seg_pos={0 0}, upload=62844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht): protocol: expected piece hash; context canceled; EOF
Sep 1 13:39:06.892072137 (71h31m3.572267706s) [seg_pos={0 0}, upload=62844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Sep 1 13:39:06.892178321 (71h31m3.57237388s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Sep 1 13:39:06.892183570 (71h31m3.572379139s) [upload=62844]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Sep 1 13:39:07.208248201 (71h31m3.88844377s) [upload=62844]: Done uploading segments...
Sep 1 13:39:07.208266486 (71h31m3.888462055s) [upload=62844]: Upload finished. err: <nil>
Duration: 7.32891711s
core
uplink
us-select-1
|