Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 897274 reports (Page 11369 of 17946)
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
2.330677532s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:47:29 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.330701568s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [......................................................X.....]
duration: 
3.537841943s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:47:14 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.537870327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
2.280937329s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:47:14 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.28096973s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................................................X.......]
duration: 
6.910158125s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:46:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.910362144s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X....XX.]
duration: 
5.845907637s
trace: 
https://jaeger.dev.storj.io/trace/4552ac32c1f0b52d
FAIL (2025-12-27 11:46:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA) context deadline exceeded
======LOGS======
Dec 27 11:45:55.476516654 (524h16m55.347599699s) [upload=459913]: Starting upload
Dec 27 11:45:55.476577920 (524h16m55.347660955s) [upload=459913]: Done starting upload
Dec 27 11:45:55.476692329 (524h16m55.347775374s) [upload=459913]: Uploading segments...
Dec 27 11:45:55.476802168 (524h16m55.347885203s) [upload=459913]: Got next segment. Inline: false
Dec 27 11:45:55.476855889 (524h16m55.347938934s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:45:55.476861030 (524h16m55.347944065s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:45:55.723875488 (524h16m55.594958543s) [seg_pos={0 0}, upload=459913]: Begin upload segment...
Dec 27 11:45:55.723960470 (524h16m55.595043515s) [seg_pos={0 0}, upload=459913]: Done begin upload segment.
Dec 27 11:45:55.723972694 (524h16m55.595055729s) [noise=true , address=145.79.227.25:4617   , piece_id=PQMBLJ4QBOGD5O3ZCQLAEFXORMS7VCEL3O23GYBAGM54HAKVMHGA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.723999244 (524h16m55.595082289s) [noise=true , address=77.90.31.12:4590     , piece_id=CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.724927191 (524h16m55.596010236s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.725287928 (524h16m55.596370973s) [noise=true , address=45.91.116.13:4154    , piece_id=WKR4R7GSSBWSQ7PSTSFREXQOPBIGTVLFOD37CFTKBSOSAWP54ZEA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.725666518 (524h16m55.596749553s) [noise=true , address=145.79.227.38:4979   , piece_id=QXJ3OOI4NNNVGFI2SOLZTBYQSMWYA7ROMSIZQWE5DBCS5PUQNM2A, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726191619 (524h16m55.597274664s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726602230 (524h16m55.597685275s) [noise=true , address=145.79.227.17:4970   , piece_id=2S2D5NFX6SMPO7IZN4KLQ5VVIRECFEXOI5UAKI7JAXYCBQNNXQEA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726985179 (524h16m55.598068254s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.727477156 (524h16m55.598560201s) [noise=true , address=145.79.227.46:4603   , piece_id=7BA7M6LQVI7ZSINJDNV7TP5FKQQS2ZAEPFXZNEQWMBQZ5BABPCKQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.728996800 (524h16m55.600079835s) [noise=true , address=145.79.227.20:4438   , piece_id=5RWZPMW2F4RRRKBQ3JO32SEWTZRYMMPSWIKX237LRI6DTIIYRJHQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.729344692 (524h16m55.600427728s) [noise=true , address=145.79.227.48:4301   , piece_id=MEYB2O3LI262QRIAT2PPXQQFINWYBIENUSNKZNT2R7IYGSOBBA5Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.730975097 (524h16m55.602058132s) [noise=true , address=77.90.31.18:4808     , piece_id=ONXVRPFO5ZQ6RO3CO6A4B3R4BM44DZ2W4ZJFFMTPZ4FYNJICN5IQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.730986308 (524h16m55.602069353s) [noise=true , address=145.79.227.11:4730   , piece_id=FLPXOH6PQI4ZDDACUXR7PIPKFDTRT7YMX5BOBAUW6DQ7XRZ3ILGA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.732810432 (524h16m55.603893467s) [noise=true , address=45.91.116.9:4262     , piece_id=LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.733198762 (524h16m55.604281807s) [noise=true , address=45.91.116.10:4084    , piece_id=QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.740289965 (524h16m55.611373s) [noise=true , address=145.79.227.15:4876   , piece_id=3UZDIWGZ4BJEUG6SV4PRTWTABCPUID3FQBK4ZMU2JGWU4VSOQ6CQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:57.446434571 (524h16m57.317517606s) [upload=459913]: Next returned nil segment
Dec 27 11:45:57.446445050 (524h16m57.317528085s) [upload=459913]: Waiting for error group managing segments...
Dec 27 11:45:57.722331565 (524h16m57.5934146s) [noise=true , address=145.79.227.17:4970   , piece_id=2S2D5NFX6SMPO7IZN4KLQ5VVIRECFEXOI5UAKI7JAXYCBQNNXQEA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.732239021 (524h16m57.603322056s) [noise=true , address=145.79.227.46:4603   , piece_id=7BA7M6LQVI7ZSINJDNV7TP5FKQQS2ZAEPFXZNEQWMBQZ5BABPCKQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.736896570 (524h16m57.607979605s) [noise=true , address=145.79.227.48:4301   , piece_id=MEYB2O3LI262QRIAT2PPXQQFINWYBIENUSNKZNT2R7IYGSOBBA5Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.737739615 (524h16m57.60882265s) [noise=true , address=145.79.227.38:4979   , piece_id=QXJ3OOI4NNNVGFI2SOLZTBYQSMWYA7ROMSIZQWE5DBCS5PUQNM2A, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.744972198 (524h16m57.616055243s) [noise=true , address=145.79.227.25:4617   , piece_id=PQMBLJ4QBOGD5O3ZCQLAEFXORMS7VCEL3O23GYBAGM54HAKVMHGA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.745509330 (524h16m57.616592375s) [noise=true , address=145.79.227.20:4438   , piece_id=5RWZPMW2F4RRRKBQ3JO32SEWTZRYMMPSWIKX237LRI6DTIIYRJHQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.747762280 (524h16m57.618845315s) [noise=true , address=145.79.227.11:4730   , piece_id=FLPXOH6PQI4ZDDACUXR7PIPKFDTRT7YMX5BOBAUW6DQ7XRZ3ILGA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.747942293 (524h16m57.619025328s) [noise=true , address=145.79.227.15:4876   , piece_id=3UZDIWGZ4BJEUG6SV4PRTWTABCPUID3FQBK4ZMU2JGWU4VSOQ6CQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.960381985 (524h16m57.831465211s) [noise=true , address=77.90.31.12:4590     , piece_id=CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.004791308 (524h16m57.875874353s) [noise=true , address=45.91.116.9:4262     , piece_id=LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.008233483 (524h16m57.879316518s) [noise=true , address=45.91.116.13:4154    , piece_id=WKR4R7GSSBWSQ7PSTSFREXQOPBIGTVLFOD37CFTKBSOSAWP54ZEA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.021138526 (524h16m57.892221571s) [noise=true , address=45.91.116.10:4084    , piece_id=QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.046604889 (524h16m57.917687924s) [noise=true , address=77.90.31.18:4808     , piece_id=ONXVRPFO5ZQ6RO3CO6A4B3R4BM44DZ2W4ZJFFMTPZ4FYNJICN5IQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.046621680 (524h16m57.917704715s) [seg_pos={0 0}, upload=459913]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:45:58.046811141 (524h16m57.917894176s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:58.046817032 (524h16m57.917900067s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047007254 (524h16m57.918090289s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:42402->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:45:58.047012886 (524h16m57.918095931s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047027383 (524h16m57.918110418s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:58.047045057 (524h16m57.918128092s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047069954 (524h16m57.918152999s) [seg_pos={0 0}, upload=459913]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:45:58.047210010 (524h16m57.918293055s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:45:58.047218677 (524h16m57.918301712s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:45:58.342313508 (524h16m58.213396543s) [upload=459913]: Done uploading segments...
Dec 27 11:45:58.342332835 (524h16m58.21341587s) [upload=459913]: Upload finished. err: <nil>
Duration: 5.845934179s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration: 
12.542939186s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:46:34 UTC) 
orbiter warning: context deadline exceeded
Duration: 12.542958752s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X....XX]
duration: 
30.00277697s
trace: 
https://jaeger.dev.storj.io/trace/4552ac32c1f0b52d
FAIL (2025-12-27 11:46:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto, Piece ID: LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA) context deadline exceeded
======LOGS======
Dec 27 11:45:55.476516654 (524h16m55.347599699s) [upload=459913]: Starting upload
Dec 27 11:45:55.476577920 (524h16m55.347660955s) [upload=459913]: Done starting upload
Dec 27 11:45:55.476692329 (524h16m55.347775374s) [upload=459913]: Uploading segments...
Dec 27 11:45:55.476802168 (524h16m55.347885203s) [upload=459913]: Got next segment. Inline: false
Dec 27 11:45:55.476855889 (524h16m55.347938934s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:45:55.476861030 (524h16m55.347944065s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:45:55.723875488 (524h16m55.594958543s) [seg_pos={0 0}, upload=459913]: Begin upload segment...
Dec 27 11:45:55.723960470 (524h16m55.595043515s) [seg_pos={0 0}, upload=459913]: Done begin upload segment.
Dec 27 11:45:55.723972694 (524h16m55.595055729s) [noise=true , address=145.79.227.25:4617   , piece_id=PQMBLJ4QBOGD5O3ZCQLAEFXORMS7VCEL3O23GYBAGM54HAKVMHGA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.723999244 (524h16m55.595082289s) [noise=true , address=77.90.31.12:4590     , piece_id=CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.724927191 (524h16m55.596010236s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.725287928 (524h16m55.596370973s) [noise=true , address=45.91.116.13:4154    , piece_id=WKR4R7GSSBWSQ7PSTSFREXQOPBIGTVLFOD37CFTKBSOSAWP54ZEA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.725666518 (524h16m55.596749553s) [noise=true , address=145.79.227.38:4979   , piece_id=QXJ3OOI4NNNVGFI2SOLZTBYQSMWYA7ROMSIZQWE5DBCS5PUQNM2A, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726191619 (524h16m55.597274664s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726602230 (524h16m55.597685275s) [noise=true , address=145.79.227.17:4970   , piece_id=2S2D5NFX6SMPO7IZN4KLQ5VVIRECFEXOI5UAKI7JAXYCBQNNXQEA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.726985179 (524h16m55.598068254s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.727477156 (524h16m55.598560201s) [noise=true , address=145.79.227.46:4603   , piece_id=7BA7M6LQVI7ZSINJDNV7TP5FKQQS2ZAEPFXZNEQWMBQZ5BABPCKQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.728996800 (524h16m55.600079835s) [noise=true , address=145.79.227.20:4438   , piece_id=5RWZPMW2F4RRRKBQ3JO32SEWTZRYMMPSWIKX237LRI6DTIIYRJHQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.729344692 (524h16m55.600427728s) [noise=true , address=145.79.227.48:4301   , piece_id=MEYB2O3LI262QRIAT2PPXQQFINWYBIENUSNKZNT2R7IYGSOBBA5Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.730975097 (524h16m55.602058132s) [noise=true , address=77.90.31.18:4808     , piece_id=ONXVRPFO5ZQ6RO3CO6A4B3R4BM44DZ2W4ZJFFMTPZ4FYNJICN5IQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.730986308 (524h16m55.602069353s) [noise=true , address=145.79.227.11:4730   , piece_id=FLPXOH6PQI4ZDDACUXR7PIPKFDTRT7YMX5BOBAUW6DQ7XRZ3ILGA, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.732810432 (524h16m55.603893467s) [noise=true , address=45.91.116.9:4262     , piece_id=LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.733198762 (524h16m55.604281807s) [noise=true , address=45.91.116.10:4084    , piece_id=QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:55.740289965 (524h16m55.611373s) [noise=true , address=145.79.227.15:4876   , piece_id=3UZDIWGZ4BJEUG6SV4PRTWTABCPUID3FQBK4ZMU2JGWU4VSOQ6CQ, seg_pos={0 0}, upload=459913]: Uploading piece...
Dec 27 11:45:57.446434571 (524h16m57.317517606s) [upload=459913]: Next returned nil segment
Dec 27 11:45:57.446445050 (524h16m57.317528085s) [upload=459913]: Waiting for error group managing segments...
Dec 27 11:45:57.722331565 (524h16m57.5934146s) [noise=true , address=145.79.227.17:4970   , piece_id=2S2D5NFX6SMPO7IZN4KLQ5VVIRECFEXOI5UAKI7JAXYCBQNNXQEA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.732239021 (524h16m57.603322056s) [noise=true , address=145.79.227.46:4603   , piece_id=7BA7M6LQVI7ZSINJDNV7TP5FKQQS2ZAEPFXZNEQWMBQZ5BABPCKQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.736896570 (524h16m57.607979605s) [noise=true , address=145.79.227.48:4301   , piece_id=MEYB2O3LI262QRIAT2PPXQQFINWYBIENUSNKZNT2R7IYGSOBBA5Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.737739615 (524h16m57.60882265s) [noise=true , address=145.79.227.38:4979   , piece_id=QXJ3OOI4NNNVGFI2SOLZTBYQSMWYA7ROMSIZQWE5DBCS5PUQNM2A, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.744972198 (524h16m57.616055243s) [noise=true , address=145.79.227.25:4617   , piece_id=PQMBLJ4QBOGD5O3ZCQLAEFXORMS7VCEL3O23GYBAGM54HAKVMHGA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.745509330 (524h16m57.616592375s) [noise=true , address=145.79.227.20:4438   , piece_id=5RWZPMW2F4RRRKBQ3JO32SEWTZRYMMPSWIKX237LRI6DTIIYRJHQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.747762280 (524h16m57.618845315s) [noise=true , address=145.79.227.11:4730   , piece_id=FLPXOH6PQI4ZDDACUXR7PIPKFDTRT7YMX5BOBAUW6DQ7XRZ3ILGA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.747942293 (524h16m57.619025328s) [noise=true , address=145.79.227.15:4876   , piece_id=3UZDIWGZ4BJEUG6SV4PRTWTABCPUID3FQBK4ZMU2JGWU4VSOQ6CQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:57.960381985 (524h16m57.831465211s) [noise=true , address=77.90.31.12:4590     , piece_id=CQTBUHKGRAGS5B5GGDOFU5FOM3CHS2U6HL3RBJEOY3NEP4PS76GA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.004791308 (524h16m57.875874353s) [noise=true , address=45.91.116.9:4262     , piece_id=LUY5BQ3DQQJRPHPPI6OMHFYZEWU25TQFC4K4TVGQFFW3GWUAPO2Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.008233483 (524h16m57.879316518s) [noise=true , address=45.91.116.13:4154    , piece_id=WKR4R7GSSBWSQ7PSTSFREXQOPBIGTVLFOD37CFTKBSOSAWP54ZEA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.021138526 (524h16m57.892221571s) [noise=true , address=45.91.116.10:4084    , piece_id=QBJCK2UFD2PWKMPCKPL3MPDNYUSLVJMPVW7IHBM7FX4SHVIZLE3A, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.046604889 (524h16m57.917687924s) [noise=true , address=77.90.31.18:4808     , piece_id=ONXVRPFO5ZQ6RO3CO6A4B3R4BM44DZ2W4ZJFFMTPZ4FYNJICN5IQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: <nil>
Dec 27 11:45:58.046621680 (524h16m57.917704715s) [seg_pos={0 0}, upload=459913]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:45:58.046811141 (524h16m57.917894176s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:58.046817032 (524h16m57.917900067s) [noise=true , address=145.79.227.30:4762   , piece_id=CBP4ZLTVN67GIOYNCZO2C7GQXRVR3P7E6JMM3L3VZDNJ6DNWUACA, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047007254 (524h16m57.918090289s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: read tcp 188.166.225.170:42402->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:45:58.047012886 (524h16m57.918095931s) [noise=true , address=45.91.116.12:4226    , piece_id=7S5QJHX3N5YYZFT5QHQN2U5QV5YMRULEFAHMVTARJATNYBUBU7UQ, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047027383 (524h16m57.918110418s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:58.047045057 (524h16m57.918128092s) [noise=true , address=45.91.116.11:4117    , piece_id=LIYVB6QVP4UA7I36FCCHLNWBMNFB5SMMORNASWBIUOSGOPJ7UB5Q, seg_pos={0 0}, upload=459913]: Long tail cancellation due to optimal threshold
Dec 27 11:45:58.047069954 (524h16m57.918152999s) [seg_pos={0 0}, upload=459913]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:45:58.047210010 (524h16m57.918293055s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:45:58.047218677 (524h16m57.918301712s) [upload=459913]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:45:58.342313508 (524h16m58.213396543s) [upload=459913]: Done uploading segments...
Dec 27 11:45:58.342332835 (524h16m58.21341587s) [upload=459913]: Upload finished. err: <nil>
Duration: 30.002788592s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.......................................................X....]
duration: 
4.365224662s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:46:23 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.365251864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
2.359455672s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:46:20 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.359479848s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [X.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X....X]
duration: 
30.003794491s
trace: 
https://jaeger.dev.storj.io/trace/3128e6ad12dc70da
FAIL (2025-12-27 11:45:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:45:25.253642029 (524h16m25.124725074s) [upload=459905]: Starting upload
Dec 27 11:45:25.253705118 (524h16m25.124788153s) [upload=459905]: Done starting upload
Dec 27 11:45:25.253789780 (524h16m25.124872825s) [upload=459905]: Uploading segments...
Dec 27 11:45:25.254016040 (524h16m25.125099095s) [upload=459905]: Got next segment. Inline: false
Dec 27 11:45:25.254075253 (524h16m25.125158288s) [upload=459905]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:45:25.254080894 (524h16m25.125163939s) [upload=459905]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:45:25.501475758 (524h16m25.372558803s) [seg_pos={0 0}, upload=459905]: Begin upload segment...
Dec 27 11:45:25.501535210 (524h16m25.372618255s) [seg_pos={0 0}, upload=459905]: Done begin upload segment.
Dec 27 11:45:25.501552263 (524h16m25.372635298s) [noise=true , address=145.79.227.10:4403   , piece_id=WUWBFCEXY3A5OXVELCUXCD7TN2R7OJAGC3TOFHODI2K75BTBCNRA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.501564175 (524h16m25.37264721s) [noise=true , address=145.79.227.14:4837   , piece_id=AE3ZRFNFMQAKZIG7FJFXSKLQTCKNHWO55EXPZS5DOUHYXCPOPFEQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.502383326 (524h16m25.373466361s) [noise=true , address=145.79.227.35:4481   , piece_id=W4KTUPVHRYO6F4AT47ICE6UA2YXYF6RZH4TK2XEOA2IO5KLCENDQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.502784910 (524h16m25.373867945s) [noise=true , address=45.91.116.7:4012     , piece_id=TK3MWCYGE52HSI3MRZ7ABI42WA5HTI77CHNJM3YCIZJL5HP6X7ZQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.503205611 (524h16m25.374288656s) [noise=true , address=145.79.227.16:4911   , piece_id=OIRJROV2R2ZR4PJNROI6I62G2S2KFAR3W5K7VQMRHVOACHVD344A, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.503644477 (524h16m25.374727512s) [noise=true , address=145.79.227.4:4185    , piece_id=V5PCIFC7FYORU4RQO4WL3JWWS3YK3NQLQJUPD5VE4VO4JIBWHIYA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.506500615 (524h16m25.37758365s) [noise=true , address=77.90.31.11:4553     , piece_id=ONWVA6T2DSEFCJVYZXETHLUSI6XXSWIPLSH3KUWCDZIN572MQLWA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.507161945 (524h16m25.37824499s) [noise=true , address=145.79.227.38:4979   , piece_id=DXFX54YUMGF2SZY2CLM6CKAISXJNRKSG6IDAKH4P74OUHGLNQRLQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.509580980 (524h16m25.380664026s) [noise=true , address=45.91.116.10:4083    , piece_id=SRINGYG6B6ZRD6UE2KMEJ55X2SJJUJHJEXKL3J5A7UPEV2LLTWDA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.511552956 (524h16m25.382635991s) [noise=true , address=77.90.31.14:4664     , piece_id=3QDG6I65C4CXFMHJ6TLKWL3UTBCFY5ZJNXJRGWFN6R75QGMMWPQA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.513161579 (524h16m25.384244624s) [noise=true , address=145.79.227.25:4618   , piece_id=XBCNBB2P7KQDLUXDG2KQGWXKMEZOCXQ43IO6VQU3GBGMWAUBFFWQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.514837329 (524h16m25.385920375s) [noise=true , address=145.79.227.13:4804   , piece_id=QGWISAVK2OCJJAHXKSSZ5FZUPUV4LX2IAO3TZ4KQK7EVMNUTHAKA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.515722365 (524h16m25.38680541s) [noise=true , address=77.90.31.17:4773     , piece_id=CBSAINRDXJPOCNRFXTKHAB2XYBMY5ZHCP4TCFVX2BAQ4G7SSQMWQ, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.517591324 (524h16m25.38867437s) [noise=true , address=145.79.227.28:4726   , piece_id=Y2HHOKB32XTDPLARXGB6AGDA5AU624DUWNR3GUIZ44E4C54CN63A, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.518979457 (524h16m25.390062502s) [noise=true , address=45.91.116.8:4048     , piece_id=JKW72DETUV4H44ZSC6J6YGZ2CNNALSGVTVEJYS6WJWKATIE2W4QA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:25.520801948 (524h16m25.391884993s) [noise=true , address=145.79.227.48:4301   , piece_id=MTNE3COM5SR4GOJO5KE2LE2PPXWUAE5SHCJA2A4QJEFOXNKEHNTA, seg_pos={0 0}, upload=459905]: Uploading piece...
Dec 27 11:45:27.225432299 (524h16m27.096515334s) [upload=459905]: Next returned nil segment
Dec 27 11:45:27.225448038 (524h16m27.096531083s) [upload=459905]: Waiting for error group managing segments...
Dec 27 11:45:27.506074009 (524h16m27.377157064s) [noise=true , address=145.79.227.35:4481   , piece_id=W4KTUPVHRYO6F4AT47ICE6UA2YXYF6RZH4TK2XEOA2IO5KLCENDQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.508981645 (524h16m27.38006468s) [noise=true , address=145.79.227.10:4403   , piece_id=WUWBFCEXY3A5OXVELCUXCD7TN2R7OJAGC3TOFHODI2K75BTBCNRA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.516027501 (524h16m27.387110536s) [noise=true , address=145.79.227.38:4979   , piece_id=DXFX54YUMGF2SZY2CLM6CKAISXJNRKSG6IDAKH4P74OUHGLNQRLQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.523261116 (524h16m27.394344151s) [noise=true , address=145.79.227.48:4301   , piece_id=MTNE3COM5SR4GOJO5KE2LE2PPXWUAE5SHCJA2A4QJEFOXNKEHNTA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.524235160 (524h16m27.395318195s) [noise=true , address=145.79.227.28:4726   , piece_id=Y2HHOKB32XTDPLARXGB6AGDA5AU624DUWNR3GUIZ44E4C54CN63A, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.524339980 (524h16m27.395423025s) [noise=true , address=145.79.227.4:4185    , piece_id=V5PCIFC7FYORU4RQO4WL3JWWS3YK3NQLQJUPD5VE4VO4JIBWHIYA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.525355173 (524h16m27.396438208s) [noise=true , address=145.79.227.14:4837   , piece_id=AE3ZRFNFMQAKZIG7FJFXSKLQTCKNHWO55EXPZS5DOUHYXCPOPFEQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.531754138 (524h16m27.402837173s) [noise=true , address=145.79.227.16:4911   , piece_id=OIRJROV2R2ZR4PJNROI6I62G2S2KFAR3W5K7VQMRHVOACHVD344A, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.533520301 (524h16m27.404603336s) [noise=true , address=145.79.227.13:4804   , piece_id=QGWISAVK2OCJJAHXKSSZ5FZUPUV4LX2IAO3TZ4KQK7EVMNUTHAKA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.545182628 (524h16m27.416265663s) [noise=true , address=145.79.227.25:4618   , piece_id=XBCNBB2P7KQDLUXDG2KQGWXKMEZOCXQ43IO6VQU3GBGMWAUBFFWQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.702490920 (524h16m27.573573955s) [noise=true , address=77.90.31.17:4773     , piece_id=CBSAINRDXJPOCNRFXTKHAB2XYBMY5ZHCP4TCFVX2BAQ4G7SSQMWQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.768848056 (524h16m27.639931101s) [noise=true , address=45.91.116.8:4048     , piece_id=JKW72DETUV4H44ZSC6J6YGZ2CNNALSGVTVEJYS6WJWKATIE2W4QA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.778873165 (524h16m27.64995621s) [noise=true , address=45.91.116.10:4083    , piece_id=SRINGYG6B6ZRD6UE2KMEJ55X2SJJUJHJEXKL3J5A7UPEV2LLTWDA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: <nil>
Dec 27 11:45:27.778897482 (524h16m27.649980527s) [seg_pos={0 0}, upload=459905]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:45:27.779308545 (524h16m27.65039159s) [noise=true , address=77.90.31.11:4553     , piece_id=ONWVA6T2DSEFCJVYZXETHLUSI6XXSWIPLSH3KUWCDZIN572MQLWA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:27.779317912 (524h16m27.650400957s) [noise=true , address=77.90.31.11:4553     , piece_id=ONWVA6T2DSEFCJVYZXETHLUSI6XXSWIPLSH3KUWCDZIN572MQLWA, seg_pos={0 0}, upload=459905]: Long tail cancellation due to optimal threshold
Dec 27 11:45:27.779401902 (524h16m27.650484937s) [noise=true , address=45.91.116.7:4012     , piece_id=TK3MWCYGE52HSI3MRZ7ABI42WA5HTI77CHNJM3YCIZJL5HP6X7ZQ, seg_pos={0 0}, upload=459905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:27.779425487 (524h16m27.650508522s) [noise=true , address=45.91.116.7:4012     , piece_id=TK3MWCYGE52HSI3MRZ7ABI42WA5HTI77CHNJM3YCIZJL5HP6X7ZQ, seg_pos={0 0}, upload=459905]: Long tail cancellation due to optimal threshold
Dec 27 11:45:27.779678088 (524h16m27.650761133s) [noise=true , address=77.90.31.14:4664     , piece_id=3QDG6I65C4CXFMHJ6TLKWL3UTBCFY5ZJNXJRGWFN6R75QGMMWPQA, seg_pos={0 0}, upload=459905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF
Dec 27 11:45:27.779688398 (524h16m27.650771433s) [noise=true , address=77.90.31.14:4664     , piece_id=3QDG6I65C4CXFMHJ6TLKWL3UTBCFY5ZJNXJRGWFN6R75QGMMWPQA, seg_pos={0 0}, upload=459905]: Long tail cancellation due to optimal threshold
Dec 27 11:45:27.779710800 (524h16m27.650793845s) [seg_pos={0 0}, upload=459905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:45:27.779832582 (524h16m27.650915618s) [upload=459905]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:45:27.779840758 (524h16m27.650923803s) [upload=459905]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:45:28.077052300 (524h16m27.948135335s) [upload=459905]: Done uploading segments...
Dec 27 11:45:28.077076476 (524h16m27.948159511s) [upload=459905]: Upload finished. err: <nil>
Duration: 30.003816513s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration: 
6.692629503s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:45:43 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.692660263s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
2.250721643s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:45:34 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.250748875s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [........................................................X...]
duration: 
3.694555201s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:45:24 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.694584368s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
2.324822875s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:45:14 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.324851911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X....]
duration: 
5.930558138s
trace: 
https://jaeger.dev.storj.io/trace/1a83b56b82486f5f
FAIL (2025-12-27 11:45:00 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q) context deadline exceeded
======LOGS======
Dec 27 11:42:06.010083771 (524h13m5.881166806s) [upload=459860]: Starting upload
Dec 27 11:42:06.010143064 (524h13m5.881226109s) [upload=459860]: Done starting upload
Dec 27 11:42:06.010334759 (524h13m5.881417794s) [upload=459860]: Uploading segments...
Dec 27 11:42:06.010372560 (524h13m5.881455606s) [upload=459860]: Got next segment. Inline: false
Dec 27 11:42:06.010399983 (524h13m5.881483018s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:42:06.010404321 (524h13m5.881487356s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:42:06.255157985 (524h13m6.12624103s) [seg_pos={0 0}, upload=459860]: Begin upload segment...
Dec 27 11:42:06.255220474 (524h13m6.12630352s) [seg_pos={0 0}, upload=459860]: Done begin upload segment.
Dec 27 11:42:06.255238969 (524h13m6.126322004s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255311026 (524h13m6.126394061s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255734983 (524h13m6.126818028s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256116380 (524h13m6.127199425s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256507255 (524h13m6.12759029s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256912215 (524h13m6.12799525s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257307408 (524h13m6.128390443s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257740803 (524h13m6.128823848s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.258895862 (524h13m6.129978898s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259262852 (524h13m6.130345897s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259965690 (524h13m6.131048725s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.260637058 (524h13m6.131720093s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.261066767 (524h13m6.132149812s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.262140260 (524h13m6.133223305s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.263597014 (524h13m6.134680049s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.265081912 (524h13m6.136164957s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:07.976845476 (524h13m7.847928511s) [upload=459860]: Next returned nil segment
Dec 27 11:42:07.976861025 (524h13m7.847944351s) [upload=459860]: Waiting for error group managing segments...
Dec 27 11:42:08.258689265 (524h13m8.12977231s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.270474637 (524h13m8.141557682s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272060456 (524h13m8.143143491s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272113477 (524h13m8.143196512s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283223683 (524h13m8.154306738s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283538422 (524h13m8.154621467s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.286122202 (524h13m8.157205237s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.292861194 (524h13m8.163944259s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.293180522 (524h13m8.164263557s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.312693749 (524h13m8.183776794s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.489075369 (524h13m8.360158414s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.572257964 (524h13m8.443340999s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626266365 (524h13m8.4973494s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626332911 (524h13m8.497415957s) [seg_pos={0 0}, upload=459860]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:42:08.626621801 (524h13m8.497704856s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection
Dec 27 11:42:08.626633404 (524h13m8.497716449s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626711312 (524h13m8.497794357s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626734896 (524h13m8.497817931s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626918827 (524h13m8.498001872s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626927854 (524h13m8.498010899s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626973321 (524h13m8.498056356s) [seg_pos={0 0}, upload=459860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:42:08.627089232 (524h13m8.498172277s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:42:08.627096646 (524h13m8.498179691s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:42:08.922984247 (524h13m8.794067302s) [upload=459860]: Done uploading segments...
Dec 27 11:42:08.923008023 (524h13m8.794091068s) [upload=459860]: Upload finished. err: <nil>
Duration: 5.930591222s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration: 
6.209804473s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:44:52 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.209874035s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [.........................................................X..]
duration: 
3.496714337s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:44:43 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.496738092s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X...]
duration: 
5.605372476s
trace: 
https://jaeger.dev.storj.io/trace/1a83b56b82486f5f
FAIL (2025-12-27 11:44:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q) context deadline exceeded
======LOGS======
Dec 27 11:42:06.010083771 (524h13m5.881166806s) [upload=459860]: Starting upload
Dec 27 11:42:06.010143064 (524h13m5.881226109s) [upload=459860]: Done starting upload
Dec 27 11:42:06.010334759 (524h13m5.881417794s) [upload=459860]: Uploading segments...
Dec 27 11:42:06.010372560 (524h13m5.881455606s) [upload=459860]: Got next segment. Inline: false
Dec 27 11:42:06.010399983 (524h13m5.881483018s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:42:06.010404321 (524h13m5.881487356s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:42:06.255157985 (524h13m6.12624103s) [seg_pos={0 0}, upload=459860]: Begin upload segment...
Dec 27 11:42:06.255220474 (524h13m6.12630352s) [seg_pos={0 0}, upload=459860]: Done begin upload segment.
Dec 27 11:42:06.255238969 (524h13m6.126322004s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255311026 (524h13m6.126394061s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255734983 (524h13m6.126818028s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256116380 (524h13m6.127199425s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256507255 (524h13m6.12759029s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256912215 (524h13m6.12799525s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257307408 (524h13m6.128390443s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257740803 (524h13m6.128823848s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.258895862 (524h13m6.129978898s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259262852 (524h13m6.130345897s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259965690 (524h13m6.131048725s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.260637058 (524h13m6.131720093s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.261066767 (524h13m6.132149812s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.262140260 (524h13m6.133223305s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.263597014 (524h13m6.134680049s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.265081912 (524h13m6.136164957s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:07.976845476 (524h13m7.847928511s) [upload=459860]: Next returned nil segment
Dec 27 11:42:07.976861025 (524h13m7.847944351s) [upload=459860]: Waiting for error group managing segments...
Dec 27 11:42:08.258689265 (524h13m8.12977231s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.270474637 (524h13m8.141557682s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272060456 (524h13m8.143143491s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272113477 (524h13m8.143196512s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283223683 (524h13m8.154306738s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283538422 (524h13m8.154621467s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.286122202 (524h13m8.157205237s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.292861194 (524h13m8.163944259s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.293180522 (524h13m8.164263557s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.312693749 (524h13m8.183776794s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.489075369 (524h13m8.360158414s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.572257964 (524h13m8.443340999s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626266365 (524h13m8.4973494s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626332911 (524h13m8.497415957s) [seg_pos={0 0}, upload=459860]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:42:08.626621801 (524h13m8.497704856s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection
Dec 27 11:42:08.626633404 (524h13m8.497716449s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626711312 (524h13m8.497794357s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626734896 (524h13m8.497817931s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626918827 (524h13m8.498001872s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626927854 (524h13m8.498010899s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626973321 (524h13m8.498056356s) [seg_pos={0 0}, upload=459860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:42:08.627089232 (524h13m8.498172277s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:42:08.627096646 (524h13m8.498179691s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:42:08.922984247 (524h13m8.794067302s) [upload=459860]: Done uploading segments...
Dec 27 11:42:08.923008023 (524h13m8.794091068s) [upload=459860]: Upload finished. err: <nil>
Duration: 5.605397655s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
6.057709817s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:44:21 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.057738402s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
2.373222496s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:44:01 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 2.373287389s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
6.207930142s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:43:56 UTC) 
orbiter warning: context deadline exceeded
Duration: 6.20795555s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [..........................................................X.]
duration: 
3.499780816s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:43:39 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.499932055s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X..]
duration: 
5.834162212s
trace: 
https://jaeger.dev.storj.io/trace/1a83b56b82486f5f
FAIL (2025-12-27 11:43:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q) context deadline exceeded
======LOGS======
Dec 27 11:42:06.010083771 (524h13m5.881166806s) [upload=459860]: Starting upload
Dec 27 11:42:06.010143064 (524h13m5.881226109s) [upload=459860]: Done starting upload
Dec 27 11:42:06.010334759 (524h13m5.881417794s) [upload=459860]: Uploading segments...
Dec 27 11:42:06.010372560 (524h13m5.881455606s) [upload=459860]: Got next segment. Inline: false
Dec 27 11:42:06.010399983 (524h13m5.881483018s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:42:06.010404321 (524h13m5.881487356s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:42:06.255157985 (524h13m6.12624103s) [seg_pos={0 0}, upload=459860]: Begin upload segment...
Dec 27 11:42:06.255220474 (524h13m6.12630352s) [seg_pos={0 0}, upload=459860]: Done begin upload segment.
Dec 27 11:42:06.255238969 (524h13m6.126322004s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255311026 (524h13m6.126394061s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255734983 (524h13m6.126818028s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256116380 (524h13m6.127199425s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256507255 (524h13m6.12759029s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256912215 (524h13m6.12799525s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257307408 (524h13m6.128390443s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257740803 (524h13m6.128823848s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.258895862 (524h13m6.129978898s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259262852 (524h13m6.130345897s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259965690 (524h13m6.131048725s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.260637058 (524h13m6.131720093s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.261066767 (524h13m6.132149812s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.262140260 (524h13m6.133223305s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.263597014 (524h13m6.134680049s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.265081912 (524h13m6.136164957s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:07.976845476 (524h13m7.847928511s) [upload=459860]: Next returned nil segment
Dec 27 11:42:07.976861025 (524h13m7.847944351s) [upload=459860]: Waiting for error group managing segments...
Dec 27 11:42:08.258689265 (524h13m8.12977231s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.270474637 (524h13m8.141557682s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272060456 (524h13m8.143143491s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272113477 (524h13m8.143196512s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283223683 (524h13m8.154306738s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283538422 (524h13m8.154621467s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.286122202 (524h13m8.157205237s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.292861194 (524h13m8.163944259s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.293180522 (524h13m8.164263557s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.312693749 (524h13m8.183776794s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.489075369 (524h13m8.360158414s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.572257964 (524h13m8.443340999s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626266365 (524h13m8.4973494s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626332911 (524h13m8.497415957s) [seg_pos={0 0}, upload=459860]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:42:08.626621801 (524h13m8.497704856s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection
Dec 27 11:42:08.626633404 (524h13m8.497716449s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626711312 (524h13m8.497794357s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626734896 (524h13m8.497817931s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626918827 (524h13m8.498001872s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626927854 (524h13m8.498010899s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626973321 (524h13m8.498056356s) [seg_pos={0 0}, upload=459860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:42:08.627089232 (524h13m8.498172277s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:42:08.627096646 (524h13m8.498179691s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:42:08.922984247 (524h13m8.794067302s) [upload=459860]: Done uploading segments...
Dec 27 11:42:08.923008023 (524h13m8.794091068s) [upload=459860]: Upload finished. err: <nil>
Duration: 5.834187411s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
25.006579259s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:43:16 UTC) 
orbiter warning: context deadline exceeded
Duration: 25.0066267s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
4.531333733s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:43:07 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.531376965s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
30.001613087s
trace: 
https://jaeger.dev.storj.io/trace/6dd289079d7aa430
FAIL (2025-12-27 11:42:57 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.0016311s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X.]
duration: 
5.821999774s
trace: 
https://jaeger.dev.storj.io/trace/1a83b56b82486f5f
FAIL (2025-12-27 11:42:56 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q) context deadline exceeded
======LOGS======
Dec 27 11:42:06.010083771 (524h13m5.881166806s) [upload=459860]: Starting upload
Dec 27 11:42:06.010143064 (524h13m5.881226109s) [upload=459860]: Done starting upload
Dec 27 11:42:06.010334759 (524h13m5.881417794s) [upload=459860]: Uploading segments...
Dec 27 11:42:06.010372560 (524h13m5.881455606s) [upload=459860]: Got next segment. Inline: false
Dec 27 11:42:06.010399983 (524h13m5.881483018s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:42:06.010404321 (524h13m5.881487356s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:42:06.255157985 (524h13m6.12624103s) [seg_pos={0 0}, upload=459860]: Begin upload segment...
Dec 27 11:42:06.255220474 (524h13m6.12630352s) [seg_pos={0 0}, upload=459860]: Done begin upload segment.
Dec 27 11:42:06.255238969 (524h13m6.126322004s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255311026 (524h13m6.126394061s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255734983 (524h13m6.126818028s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256116380 (524h13m6.127199425s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256507255 (524h13m6.12759029s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256912215 (524h13m6.12799525s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257307408 (524h13m6.128390443s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257740803 (524h13m6.128823848s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.258895862 (524h13m6.129978898s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259262852 (524h13m6.130345897s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259965690 (524h13m6.131048725s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.260637058 (524h13m6.131720093s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.261066767 (524h13m6.132149812s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.262140260 (524h13m6.133223305s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.263597014 (524h13m6.134680049s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.265081912 (524h13m6.136164957s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:07.976845476 (524h13m7.847928511s) [upload=459860]: Next returned nil segment
Dec 27 11:42:07.976861025 (524h13m7.847944351s) [upload=459860]: Waiting for error group managing segments...
Dec 27 11:42:08.258689265 (524h13m8.12977231s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.270474637 (524h13m8.141557682s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272060456 (524h13m8.143143491s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272113477 (524h13m8.143196512s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283223683 (524h13m8.154306738s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283538422 (524h13m8.154621467s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.286122202 (524h13m8.157205237s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.292861194 (524h13m8.163944259s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.293180522 (524h13m8.164263557s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.312693749 (524h13m8.183776794s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.489075369 (524h13m8.360158414s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.572257964 (524h13m8.443340999s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626266365 (524h13m8.4973494s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626332911 (524h13m8.497415957s) [seg_pos={0 0}, upload=459860]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:42:08.626621801 (524h13m8.497704856s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection
Dec 27 11:42:08.626633404 (524h13m8.497716449s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626711312 (524h13m8.497794357s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626734896 (524h13m8.497817931s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626918827 (524h13m8.498001872s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626927854 (524h13m8.498010899s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626973321 (524h13m8.498056356s) [seg_pos={0 0}, upload=459860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:42:08.627089232 (524h13m8.498172277s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:42:08.627096646 (524h13m8.498179691s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:42:08.922984247 (524h13m8.794067302s) [upload=459860]: Done uploading segments...
Dec 27 11:42:08.923008023 (524h13m8.794091068s) [upload=459860]: Upload finished. err: <nil>
Duration: 5.822026425s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
30.001152788s
trace: 
https://jaeger.dev.storj.io/trace/578b4df2b1bacee
FAIL (2025-12-27 11:42:45 UTC) 
orbiter warning: context deadline exceeded
Duration: 30.001174621s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [..X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.X]
duration: 
30.002530174s
trace: 
https://jaeger.dev.storj.io/trace/1a83b56b82486f5f
FAIL (2025-12-27 11:42:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: 4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA) context deadline exceeded; piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q) context deadline exceeded
======LOGS======
Dec 27 11:42:06.010083771 (524h13m5.881166806s) [upload=459860]: Starting upload
Dec 27 11:42:06.010143064 (524h13m5.881226109s) [upload=459860]: Done starting upload
Dec 27 11:42:06.010334759 (524h13m5.881417794s) [upload=459860]: Uploading segments...
Dec 27 11:42:06.010372560 (524h13m5.881455606s) [upload=459860]: Got next segment. Inline: false
Dec 27 11:42:06.010399983 (524h13m5.881483018s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:42:06.010404321 (524h13m5.881487356s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:42:06.255157985 (524h13m6.12624103s) [seg_pos={0 0}, upload=459860]: Begin upload segment...
Dec 27 11:42:06.255220474 (524h13m6.12630352s) [seg_pos={0 0}, upload=459860]: Done begin upload segment.
Dec 27 11:42:06.255238969 (524h13m6.126322004s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255311026 (524h13m6.126394061s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.255734983 (524h13m6.126818028s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256116380 (524h13m6.127199425s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256507255 (524h13m6.12759029s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.256912215 (524h13m6.12799525s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257307408 (524h13m6.128390443s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.257740803 (524h13m6.128823848s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.258895862 (524h13m6.129978898s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259262852 (524h13m6.130345897s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.259965690 (524h13m6.131048725s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.260637058 (524h13m6.131720093s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.261066767 (524h13m6.132149812s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.262140260 (524h13m6.133223305s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.263597014 (524h13m6.134680049s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:06.265081912 (524h13m6.136164957s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Uploading piece...
Dec 27 11:42:07.976845476 (524h13m7.847928511s) [upload=459860]: Next returned nil segment
Dec 27 11:42:07.976861025 (524h13m7.847944351s) [upload=459860]: Waiting for error group managing segments...
Dec 27 11:42:08.258689265 (524h13m8.12977231s) [noise=true , address=145.79.227.51:4411   , piece_id=6HYLTSSPL53RJ2SQHAEZGKCFMXHUTNPCQKCZZFESEQYQYUZG3ZAA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.270474637 (524h13m8.141557682s) [noise=true , address=145.79.227.49:4338   , piece_id=4LAPJYQVX5GNZLI2DXEDVHVBEW2QYDP2WYLSDPQTH4FKWSA7ZAXQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272060456 (524h13m8.143143491s) [noise=true , address=145.79.227.10:4402   , piece_id=HL7R3SCTEAFL7U3R67M743ZO3O2NBWXVNO4GIEFT2XL2T6W7ZF3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.272113477 (524h13m8.143196512s) [noise=true , address=145.79.227.16:4911   , piece_id=WH3SXFKZETNGCEXFDOLN6XIIRQDC3GKGQXTRG4ODF7Z57BBWMJCA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283223683 (524h13m8.154306738s) [noise=true , address=145.79.227.3:4149    , piece_id=OUEYW5E746KSMYNDUNQE47CNYMKX5FG2JSXB75V32INKCXCEJN2A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.283538422 (524h13m8.154621467s) [noise=true , address=145.79.227.52:4447   , piece_id=YPLS5DPALVKSRA6JMPZYE6EBFAATMGFMWF22IGAQPN6O7FAOTL3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.286122202 (524h13m8.157205237s) [noise=true , address=145.79.227.43:4157   , piece_id=MGYCBFS4ZWNPWF5K3STWA5V4CHCYQ7CMDXJAGUH3POQAAFA6XLFA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.292861194 (524h13m8.163944259s) [noise=true , address=145.79.227.2:4114    , piece_id=IAAHV57P5ANLMM5OAD2WASBWC7UGXXRMPNPIQS7ZUQE5NLHI3EKA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.293180522 (524h13m8.164263557s) [noise=true , address=145.79.227.38:4979   , piece_id=ZCKBCKVW2TWXEJVOFQA47GNT4K7QDOFBGXMAD7CC7LPKI5EBDB3A, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.312693749 (524h13m8.183776794s) [noise=true , address=145.79.227.47:4265   , piece_id=HY772CQY65CEFVDC2BEGA4GCQC5VNSXPRTKNZRJOR2PV6WQP256Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.489075369 (524h13m8.360158414s) [noise=true , address=45.91.116.4:4937     , piece_id=BQDY6VDLO7WLIM65SHXNSSOUXQN4DDYKKJHV4U47OFOOULBP332Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.572257964 (524h13m8.443340999s) [noise=true , address=45.91.116.6:4975     , piece_id=4VYELUPPPJD5OS6AAZCKK3JEHNCB3X6BVBJJDHIMCLAIKFCHLCLA, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626266365 (524h13m8.4973494s) [noise=true , address=45.91.116.5:4299     , piece_id=AW226SSWALDXRY25V57D4GOUH53EWRRRPVEJINDA4CHY7BELGW2Q, seg_pos={0 0}, upload=459860]: Done uploading piece. err: <nil>
Dec 27 11:42:08.626332911 (524h13m8.497415957s) [seg_pos={0 0}, upload=459860]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:42:08.626621801 (524h13m8.497704856s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46440->45.91.116.13:4155: use of closed network connection
Dec 27 11:42:08.626633404 (524h13m8.497716449s) [noise=true , address=45.91.116.13:4155    , piece_id=QNER6DZALSH6AWU5OZRL43Z7SHJIAOV76WHC76RG5ZC4NTUHAOKQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626711312 (524h13m8.497794357s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aik37v1FXxLLPx5EoA5EWszHVsRUz7kAriBYUbWEtN15555xE): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626734896 (524h13m8.497817931s) [noise=true , address=145.79.227.30:4761   , piece_id=5SCFFMH6BHJCTJ7Z5OVMDKTZAJDYINWY3BS7XE53YIH6XBATODJQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626918827 (524h13m8.498001872s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Dec 27 11:42:08.626927854 (524h13m8.498010899s) [noise=true , address=45.91.116.14:4189    , piece_id=YPL4LL4PDEKEAUHWW4W3B6U53E4BYGO5A63WGELIL7JSMSGEP3ZQ, seg_pos={0 0}, upload=459860]: Long tail cancellation due to optimal threshold
Dec 27 11:42:08.626973321 (524h13m8.498056356s) [seg_pos={0 0}, upload=459860]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:42:08.627089232 (524h13m8.498172277s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:42:08.627096646 (524h13m8.498179691s) [upload=459860]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:42:08.922984247 (524h13m8.794067302s) [upload=459860]: Done uploading segments...
Dec 27 11:42:08.923008023 (524h13m8.794091068s) [upload=459860]: Upload finished. err: <nil>
Duration: 30.002556513s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
1 out of 60 is failed (threshold=6) [...........................................................X]
duration: 
5.001046877s
trace: 
https://jaeger.dev.storj.io/trace/12569321705fed16
FAIL (2025-12-27 11:42:31 UTC) 
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001064962s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [...X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX.]
duration: 
5.942519204s
trace: 
https://jaeger.dev.storj.io/trace/3ae7b29dbab722b9
FAIL (2025-12-27 11:41:07 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA) context deadline exceeded
======LOGS======
=459831]: Starting upload
Dec 27 11:40:00.608240287 (524h11m0.479323342s) [upload=459831]: Done starting upload
Dec 27 11:40:00.608336400 (524h11m0.479419455s) [upload=459831]: Uploading segments...
Dec 27 11:40:00.608648484 (524h11m0.479731529s) [upload=459831]: Got next segment. Inline: false
Dec 27 11:40:00.608687078 (524h11m0.479770123s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:40:00.608692668 (524h11m0.479775703s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:40:00.850922135 (524h11m0.72200517s) [seg_pos={0 0}, upload=459831]: Begin upload segment...
Dec 27 11:40:00.850968784 (524h11m0.722051819s) [seg_pos={0 0}, upload=459831]: Done begin upload segment.
Dec 27 11:40:00.850987320 (524h11m0.722070365s) [noise=true , address=45.91.116.11:4118    , piece_id=PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.851056511 (524h11m0.722139556s) [noise=true , address=145.79.227.37:4566   , piece_id=TJELAKPRYJ5LZXNFW3OWPEZF4UVRWPPGKY2DFPYBFR57JJSC3TPA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.851914725 (524h11m0.72299777s) [noise=true , address=145.79.227.29:4942   , piece_id=OBLIGMUDQW4OJYY6C4V7BLKQKS5J525QRCGOGRHTOE2VUCT35EIA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.852450556 (524h11m0.723533601s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.852880695 (524h11m0.72396374s) [noise=true , address=145.79.227.28:4726   , piece_id=PSO7TZGG6AHBDZOTR7YIAQ3MKLUGTCJK6LAAK67PTIQIH5WLM57Q, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853321715 (524h11m0.72440475s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853767332 (524h11m0.724850367s) [noise=true , address=145.79.227.17:4972   , piece_id=YJGVTXF6V22JZQYFKIG2R4MOEUK7E4RLNVTTWLF3HNRVUO5RDNEA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853983505 (524h11m0.72506656s) [noise=true , address=145.79.227.50:4373   , piece_id=XYC2D6NW73X4AXAFSXKWDYWZZ2OQOR4MCTALGKCIAAVNZVQPZ6YA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.854519826 (524h11m0.725602871s) [noise=true , address=145.79.227.35:4483   , piece_id=AB4KIL22NU2F3ABTBHKJ3FO6LFGBNL66L5PZSNMOHALJZJN4QITQ, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855024838 (524h11m0.726107873s) [noise=true , address=145.79.227.11:4730   , piece_id=YH3C3ULPVVVHBXTDYAUQXHBKDB42F7246IKYNSSDNZ2O5MTF5T3A, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855319789 (524h11m0.726402824s) [noise=true , address=77.90.31.21:4880     , piece_id=R6UL2GOI4P6NE7SID2LSOFIW6MTBJN6ZM5YKHV4IH3YHM6WADAIQ, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855492368 (524h11m0.726575433s) [noise=true , address=77.90.31.17:4770     , piece_id=AFSLRORA6XL226SQTKYIAIWMEZJEKWT7YNOXDF3LRVKJPBSS25LA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855957293 (524h11m0.727040328s) [noise=true , address=145.79.227.7:4294    , piece_id=DF5NU3J3KQ3CYBKDLXCDWQHOHN36PA3U7RZ4PAGCV4H5J65KNSHA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.856490789 (524h11m0.727573834s) [noise=true , address=45.91.116.12:4226    , piece_id=ICTPMZXE7V5GA6YYOUQZFATMEWH4HKFV2YMQVB42SHLKVMZ3MV6A, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.857684913 (524h11m0.728767948s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.859073597 (524h11m0.730156632s) [noise=true , address=45.91.116.8:4048     , piece_id=JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:02.485536082 (524h11m2.356619137s) [upload=459831]: Next returned nil segment
Dec 27 11:40:02.485557443 (524h11m2.356640488s) [upload=459831]: Waiting for error group managing segments...
Dec 27 11:40:02.884188696 (524h11m2.755271741s) [noise=true , address=145.79.227.50:4373   , piece_id=XYC2D6NW73X4AXAFSXKWDYWZZ2OQOR4MCTALGKCIAAVNZVQPZ6YA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.885177419 (524h11m2.756260464s) [noise=true , address=145.79.227.11:4730   , piece_id=YH3C3ULPVVVHBXTDYAUQXHBKDB42F7246IKYNSSDNZ2O5MTF5T3A, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.886468527 (524h11m2.757551573s) [noise=true , address=145.79.227.7:4294    , piece_id=DF5NU3J3KQ3CYBKDLXCDWQHOHN36PA3U7RZ4PAGCV4H5J65KNSHA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.890405935 (524h11m2.76148899s) [noise=true , address=145.79.227.17:4972   , piece_id=YJGVTXF6V22JZQYFKIG2R4MOEUK7E4RLNVTTWLF3HNRVUO5RDNEA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.900264958 (524h11m2.771348003s) [noise=true , address=145.79.227.37:4566   , piece_id=TJELAKPRYJ5LZXNFW3OWPEZF4UVRWPPGKY2DFPYBFR57JJSC3TPA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.902929833 (524h11m2.774012878s) [noise=true , address=145.79.227.28:4726   , piece_id=PSO7TZGG6AHBDZOTR7YIAQ3MKLUGTCJK6LAAK67PTIQIH5WLM57Q, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.903821400 (524h11m2.774904435s) [noise=true , address=145.79.227.35:4483   , piece_id=AB4KIL22NU2F3ABTBHKJ3FO6LFGBNL66L5PZSNMOHALJZJN4QITQ, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.943147374 (524h11m2.814230429s) [noise=true , address=145.79.227.29:4942   , piece_id=OBLIGMUDQW4OJYY6C4V7BLKQKS5J525QRCGOGRHTOE2VUCT35EIA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.062744558 (524h11m2.933827613s) [noise=true , address=77.90.31.17:4770     , piece_id=AFSLRORA6XL226SQTKYIAIWMEZJEKWT7YNOXDF3LRVKJPBSS25LA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.209632838 (524h11m3.080715883s) [noise=true , address=45.91.116.12:4226    , piece_id=ICTPMZXE7V5GA6YYOUQZFATMEWH4HKFV2YMQVB42SHLKVMZ3MV6A, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.216380567 (524h11m3.087463612s) [noise=true , address=45.91.116.8:4048     , piece_id=JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.223700836 (524h11m3.094783881s) [noise=true , address=77.90.31.21:4880     , piece_id=R6UL2GOI4P6NE7SID2LSOFIW6MTBJN6ZM5YKHV4IH3YHM6WADAIQ, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.232277778 (524h11m3.103360823s) [noise=true , address=45.91.116.11:4118    , piece_id=PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.232312995 (524h11m3.10339604s) [seg_pos={0 0}, upload=459831]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:40:03.232614149 (524h11m3.103697204s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 11:40:03.232625149 (524h11m3.103708195s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232689502 (524h11m3.103772557s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection
Dec 27 11:40:03.232724469 (524h11m3.103807534s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232896717 (524h11m3.103979772s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:35338->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:40:03.232907878 (524h11m3.103990923s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232935130 (524h11m3.104018175s) [seg_pos={0 0}, upload=459831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:40:03.233033417 (524h11m3.104116472s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:40:03.233041883 (524h11m3.104124928s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:40:03.568859939 (524h11m3.439942984s) [upload=459831]: Done uploading segments...
Dec 27 11:40:03.568888363 (524h11m3.439971408s) [upload=459831]: Upload finished. err: <nil>
Duration: 5.942546947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [....X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....XX]
duration: 
30.003539124s
trace: 
https://jaeger.dev.storj.io/trace/3ae7b29dbab722b9
FAIL (2025-12-27 11:40:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q) context deadline exceeded; piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA) context deadline exceeded
======LOGS======
=459831]: Starting upload
Dec 27 11:40:00.608240287 (524h11m0.479323342s) [upload=459831]: Done starting upload
Dec 27 11:40:00.608336400 (524h11m0.479419455s) [upload=459831]: Uploading segments...
Dec 27 11:40:00.608648484 (524h11m0.479731529s) [upload=459831]: Got next segment. Inline: false
Dec 27 11:40:00.608687078 (524h11m0.479770123s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:40:00.608692668 (524h11m0.479775703s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:40:00.850922135 (524h11m0.72200517s) [seg_pos={0 0}, upload=459831]: Begin upload segment...
Dec 27 11:40:00.850968784 (524h11m0.722051819s) [seg_pos={0 0}, upload=459831]: Done begin upload segment.
Dec 27 11:40:00.850987320 (524h11m0.722070365s) [noise=true , address=45.91.116.11:4118    , piece_id=PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.851056511 (524h11m0.722139556s) [noise=true , address=145.79.227.37:4566   , piece_id=TJELAKPRYJ5LZXNFW3OWPEZF4UVRWPPGKY2DFPYBFR57JJSC3TPA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.851914725 (524h11m0.72299777s) [noise=true , address=145.79.227.29:4942   , piece_id=OBLIGMUDQW4OJYY6C4V7BLKQKS5J525QRCGOGRHTOE2VUCT35EIA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.852450556 (524h11m0.723533601s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.852880695 (524h11m0.72396374s) [noise=true , address=145.79.227.28:4726   , piece_id=PSO7TZGG6AHBDZOTR7YIAQ3MKLUGTCJK6LAAK67PTIQIH5WLM57Q, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853321715 (524h11m0.72440475s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853767332 (524h11m0.724850367s) [noise=true , address=145.79.227.17:4972   , piece_id=YJGVTXF6V22JZQYFKIG2R4MOEUK7E4RLNVTTWLF3HNRVUO5RDNEA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.853983505 (524h11m0.72506656s) [noise=true , address=145.79.227.50:4373   , piece_id=XYC2D6NW73X4AXAFSXKWDYWZZ2OQOR4MCTALGKCIAAVNZVQPZ6YA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.854519826 (524h11m0.725602871s) [noise=true , address=145.79.227.35:4483   , piece_id=AB4KIL22NU2F3ABTBHKJ3FO6LFGBNL66L5PZSNMOHALJZJN4QITQ, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855024838 (524h11m0.726107873s) [noise=true , address=145.79.227.11:4730   , piece_id=YH3C3ULPVVVHBXTDYAUQXHBKDB42F7246IKYNSSDNZ2O5MTF5T3A, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855319789 (524h11m0.726402824s) [noise=true , address=77.90.31.21:4880     , piece_id=R6UL2GOI4P6NE7SID2LSOFIW6MTBJN6ZM5YKHV4IH3YHM6WADAIQ, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855492368 (524h11m0.726575433s) [noise=true , address=77.90.31.17:4770     , piece_id=AFSLRORA6XL226SQTKYIAIWMEZJEKWT7YNOXDF3LRVKJPBSS25LA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.855957293 (524h11m0.727040328s) [noise=true , address=145.79.227.7:4294    , piece_id=DF5NU3J3KQ3CYBKDLXCDWQHOHN36PA3U7RZ4PAGCV4H5J65KNSHA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.856490789 (524h11m0.727573834s) [noise=true , address=45.91.116.12:4226    , piece_id=ICTPMZXE7V5GA6YYOUQZFATMEWH4HKFV2YMQVB42SHLKVMZ3MV6A, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.857684913 (524h11m0.728767948s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:00.859073597 (524h11m0.730156632s) [noise=true , address=45.91.116.8:4048     , piece_id=JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q, seg_pos={0 0}, upload=459831]: Uploading piece...
Dec 27 11:40:02.485536082 (524h11m2.356619137s) [upload=459831]: Next returned nil segment
Dec 27 11:40:02.485557443 (524h11m2.356640488s) [upload=459831]: Waiting for error group managing segments...
Dec 27 11:40:02.884188696 (524h11m2.755271741s) [noise=true , address=145.79.227.50:4373   , piece_id=XYC2D6NW73X4AXAFSXKWDYWZZ2OQOR4MCTALGKCIAAVNZVQPZ6YA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.885177419 (524h11m2.756260464s) [noise=true , address=145.79.227.11:4730   , piece_id=YH3C3ULPVVVHBXTDYAUQXHBKDB42F7246IKYNSSDNZ2O5MTF5T3A, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.886468527 (524h11m2.757551573s) [noise=true , address=145.79.227.7:4294    , piece_id=DF5NU3J3KQ3CYBKDLXCDWQHOHN36PA3U7RZ4PAGCV4H5J65KNSHA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.890405935 (524h11m2.76148899s) [noise=true , address=145.79.227.17:4972   , piece_id=YJGVTXF6V22JZQYFKIG2R4MOEUK7E4RLNVTTWLF3HNRVUO5RDNEA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.900264958 (524h11m2.771348003s) [noise=true , address=145.79.227.37:4566   , piece_id=TJELAKPRYJ5LZXNFW3OWPEZF4UVRWPPGKY2DFPYBFR57JJSC3TPA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.902929833 (524h11m2.774012878s) [noise=true , address=145.79.227.28:4726   , piece_id=PSO7TZGG6AHBDZOTR7YIAQ3MKLUGTCJK6LAAK67PTIQIH5WLM57Q, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.903821400 (524h11m2.774904435s) [noise=true , address=145.79.227.35:4483   , piece_id=AB4KIL22NU2F3ABTBHKJ3FO6LFGBNL66L5PZSNMOHALJZJN4QITQ, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:02.943147374 (524h11m2.814230429s) [noise=true , address=145.79.227.29:4942   , piece_id=OBLIGMUDQW4OJYY6C4V7BLKQKS5J525QRCGOGRHTOE2VUCT35EIA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.062744558 (524h11m2.933827613s) [noise=true , address=77.90.31.17:4770     , piece_id=AFSLRORA6XL226SQTKYIAIWMEZJEKWT7YNOXDF3LRVKJPBSS25LA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.209632838 (524h11m3.080715883s) [noise=true , address=45.91.116.12:4226    , piece_id=ICTPMZXE7V5GA6YYOUQZFATMEWH4HKFV2YMQVB42SHLKVMZ3MV6A, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.216380567 (524h11m3.087463612s) [noise=true , address=45.91.116.8:4048     , piece_id=JGQG2NLTQXIUXAZRA5DBLNM2GAUYA2LRNJSMM45WVIBLL225AK6Q, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.223700836 (524h11m3.094783881s) [noise=true , address=77.90.31.21:4880     , piece_id=R6UL2GOI4P6NE7SID2LSOFIW6MTBJN6ZM5YKHV4IH3YHM6WADAIQ, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.232277778 (524h11m3.103360823s) [noise=true , address=45.91.116.11:4118    , piece_id=PCMHW5N75BS7FCYJTYE3E3E44AFKJPYG6JELQQWBWUVHINS6APKA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: <nil>
Dec 27 11:40:03.232312995 (524h11m3.10339604s) [seg_pos={0 0}, upload=459831]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:40:03.232614149 (524h11m3.103697204s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 11:40:03.232625149 (524h11m3.103708195s) [noise=true , address=45.91.116.6:4973     , piece_id=FA45CNGEEB2ZP6R3KRRE5TI2HBL3A2LKGXFFTVLYYRIA4OKZPXTA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232689502 (524h11m3.103772557s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection; write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42170->45.91.116.4:4939: use of closed network connection
Dec 27 11:40:03.232724469 (524h11m3.103807534s) [noise=true , address=45.91.116.4:4939     , piece_id=RY55HF6OLMSDU7LHPOA6Z2JWSI4N2GFVVTAH35RX6WCDABBNF3GA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232896717 (524h11m3.103979772s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: read tcp 188.166.225.170:35338->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:40:03.232907878 (524h11m3.103990923s) [noise=true , address=45.91.116.13:4154    , piece_id=JCEEVKDXIFIHG4AU5OVSKIUWALBHRWJT7XR23ZRNOMLAYHEL5NKA, seg_pos={0 0}, upload=459831]: Long tail cancellation due to optimal threshold
Dec 27 11:40:03.232935130 (524h11m3.104018175s) [seg_pos={0 0}, upload=459831]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:40:03.233033417 (524h11m3.104116472s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:40:03.233041883 (524h11m3.104124928s) [upload=459831]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:40:03.568859939 (524h11m3.439942984s) [upload=459831]: Done uploading segments...
Dec 27 11:40:03.568888363 (524h11m3.439971408s) [upload=459831]: Upload finished. err: <nil>
Duration: 30.003558351s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [.....X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....X]
duration: 
30.002417336s
trace: 
https://jaeger.dev.storj.io/trace/3607df1c8153fa64
FAIL (2025-12-27 11:39:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: YNK4JBLDH7UU66MUBCALUED4KXC5RTZ65X63AEDTM42DB4MJHWBQ) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: SFOXR3LTDIR5YWHXQI4IOW5WC6EW3YRWMZ5TCFT2GY7ZZSE5RKAQ) context deadline exceeded; piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: LJ4PIDC56IUFTPSWPMPO4F33NWWH37KKVWW2BDE452SKRLQCRYIQ) context deadline exceeded
======LOGS======
Dec 27 11:39:28.175591429 (524h10m28.046674464s) [upload=459820]: Starting upload
Dec 27 11:39:28.175713792 (524h10m28.046796837s) [upload=459820]: Done starting upload
Dec 27 11:39:28.175782884 (524h10m28.046865929s) [upload=459820]: Uploading segments...
Dec 27 11:39:28.175910707 (524h10m28.046993742s) [upload=459820]: Got next segment. Inline: false
Dec 27 11:39:28.175956544 (524h10m28.047039579s) [upload=459820]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:39:28.176009435 (524h10m28.04709247s) [upload=459820]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:39:28.432880713 (524h10m28.303963748s) [seg_pos={0 0}, upload=459820]: Begin upload segment...
Dec 27 11:39:28.432939835 (524h10m28.304022881s) [seg_pos={0 0}, upload=459820]: Done begin upload segment.
Dec 27 11:39:28.432952469 (524h10m28.304035504s) [noise=true , address=45.91.116.12:4227    , piece_id=YNK4JBLDH7UU66MUBCALUED4KXC5RTZ65X63AEDTM42DB4MJHWBQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.432965615 (524h10m28.30404865s) [noise=true , address=145.79.227.52:4447   , piece_id=HQIY24PBCIO6UELHBEICC63ZYMTN4A7FQKC3AR4KVTKQMHOELDJA, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.433479484 (524h10m28.304562519s) [noise=true , address=45.91.116.14:4189    , piece_id=SFOXR3LTDIR5YWHXQI4IOW5WC6EW3YRWMZ5TCFT2GY7ZZSE5RKAQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.433610863 (524h10m28.304693898s) [noise=true , address=145.79.227.2:4113    , piece_id=JT7WR7AEKBUHVTSJGG7SN6BZC732B6DTXRFQISPR5AE6G6BDAZGA, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.433933538 (524h10m28.305016573s) [noise=true , address=77.90.31.10:4520     , piece_id=HWRLLKDWNPYKOHONGFPD6NDS74NUZM6QKSAI2FOX45IEWJGLYXKQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.434263336 (524h10m28.305346381s) [noise=true , address=77.90.31.15:4700     , piece_id=2N7ZK2YFNLJ2REQSAUPHAWWAEL3N7Y4VNFU4YQAERK7KH52PKFHA, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.434524974 (524h10m28.305608019s) [noise=true , address=145.79.227.37:4567   , piece_id=2YVG2SRNHSEB5XGS6O6BJHKMYAUZUEEQABGEO7V2Y6Y52U43ZN3A, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.434672405 (524h10m28.30575544s) [noise=true , address=45.91.116.8:4047     , piece_id=BVFXKD3BCD23UUYBWRLTVFS3VVMYJIMNVXD4THNGEYP26PDKBC2Q, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.435173248 (524h10m28.306256283s) [noise=true , address=45.91.116.7:4010     , piece_id=7WLIH23HNUGGBEHPQTF57FSLCPCLDLZU6MKQMC6WM5H2F6VXYXHQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.435368049 (524h10m28.306451084s) [noise=true , address=77.90.31.21:4880     , piece_id=LJ4PIDC56IUFTPSWPMPO4F33NWWH37KKVWW2BDE452SKRLQCRYIQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.435665044 (524h10m28.30674809s) [noise=true , address=145.79.227.45:4229   , piece_id=XXOJYLWVDYLLITHS5T3EUCQYHIUVUZZPDJF7IGVL7KS5F6QLNWUQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.435923277 (524h10m28.307006312s) [noise=true , address=145.79.227.50:4375   , piece_id=QOXQGMTBLYK4RSJT7GJXZRMXMTF3ILNUXNVIIXZEBL2NKIZBQBNQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.436273794 (524h10m28.307356839s) [noise=true , address=45.91.116.5:4298     , piece_id=25TN46SZQINXNMJBQAUHQPELKJIRXHAG5YRKGR2AWHZ6WOS2WH5Q, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.436580969 (524h10m28.307664004s) [noise=true , address=145.79.227.42:4122   , piece_id=NLLGRJALXEDWNPYZW222BILXVBYOUPHUAWBADBTFL45OACH5UUWQ, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.437074349 (524h10m28.308157384s) [noise=true , address=145.79.227.40:4050   , piece_id=TFPMHYGYSLQGMCRUAKP4C6IXDGKOLSFMJCR33QNTILJXH5IMJ3OA, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:28.437525047 (524h10m28.308608082s) [noise=true , address=77.90.31.19:4844     , piece_id=6BUZKQZSYQKWNHTI7YUBIQMXCRCLLXFYXCOW45NLQVLABQMGQ4WA, seg_pos={0 0}, upload=459820]: Uploading piece...
Dec 27 11:39:30.156671098 (524h10m30.027754133s) [upload=459820]: Next returned nil segment
Dec 27 11:39:30.156705484 (524h10m30.027788519s) [upload=459820]: Waiting for error group managing segments...
Dec 27 11:39:30.441236971 (524h10m30.312320036s) [noise=true , address=145.79.227.42:4122   , piece_id=NLLGRJALXEDWNPYZW222BILXVBYOUPHUAWBADBTFL45OACH5UUWQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.445722823 (524h10m30.316805858s) [noise=true , address=145.79.227.52:4447   , piece_id=HQIY24PBCIO6UELHBEICC63ZYMTN4A7FQKC3AR4KVTKQMHOELDJA, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.448419397 (524h10m30.319502442s) [noise=true , address=145.79.227.45:4229   , piece_id=XXOJYLWVDYLLITHS5T3EUCQYHIUVUZZPDJF7IGVL7KS5F6QLNWUQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.461505215 (524h10m30.33258825s) [noise=true , address=145.79.227.2:4113    , piece_id=JT7WR7AEKBUHVTSJGG7SN6BZC732B6DTXRFQISPR5AE6G6BDAZGA, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.465690053 (524h10m30.336773099s) [noise=true , address=145.79.227.37:4567   , piece_id=2YVG2SRNHSEB5XGS6O6BJHKMYAUZUEEQABGEO7V2Y6Y52U43ZN3A, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.466079815 (524h10m30.33716286s) [noise=true , address=145.79.227.50:4375   , piece_id=QOXQGMTBLYK4RSJT7GJXZRMXMTF3ILNUXNVIIXZEBL2NKIZBQBNQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.641026755 (524h10m30.5121098s) [noise=true , address=77.90.31.21:4880     , piece_id=LJ4PIDC56IUFTPSWPMPO4F33NWWH37KKVWW2BDE452SKRLQCRYIQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.667757084 (524h10m30.538840129s) [noise=true , address=45.91.116.14:4189    , piece_id=SFOXR3LTDIR5YWHXQI4IOW5WC6EW3YRWMZ5TCFT2GY7ZZSE5RKAQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.798425829 (524h10m30.669508864s) [noise=true , address=45.91.116.5:4298     , piece_id=25TN46SZQINXNMJBQAUHQPELKJIRXHAG5YRKGR2AWHZ6WOS2WH5Q, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.818438768 (524h10m30.689521803s) [noise=true , address=77.90.31.19:4844     , piece_id=6BUZKQZSYQKWNHTI7YUBIQMXCRCLLXFYXCOW45NLQVLABQMGQ4WA, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:30.836289749 (524h10m30.707372784s) [noise=true , address=77.90.31.10:4520     , piece_id=HWRLLKDWNPYKOHONGFPD6NDS74NUZM6QKSAI2FOX45IEWJGLYXKQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:31.477528729 (524h10m31.348611774s) [noise=true , address=45.91.116.8:4047     , piece_id=BVFXKD3BCD23UUYBWRLTVFS3VVMYJIMNVXD4THNGEYP26PDKBC2Q, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:31.570978133 (524h10m31.442061178s) [noise=true , address=45.91.116.12:4227    , piece_id=YNK4JBLDH7UU66MUBCALUED4KXC5RTZ65X63AEDTM42DB4MJHWBQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: <nil>
Dec 27 11:39:31.570996738 (524h10m31.442079783s) [seg_pos={0 0}, upload=459820]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:39:31.571225084 (524h10m31.442308129s) [noise=true , address=145.79.227.40:4050   , piece_id=TFPMHYGYSLQGMCRUAKP4C6IXDGKOLSFMJCR33QNTILJXH5IMJ3OA, seg_pos={0 0}, upload=459820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF
Dec 27 11:39:31.571233038 (524h10m31.442316084s) [noise=true , address=145.79.227.40:4050   , piece_id=TFPMHYGYSLQGMCRUAKP4C6IXDGKOLSFMJCR33QNTILJXH5IMJ3OA, seg_pos={0 0}, upload=459820]: Long tail cancellation due to optimal threshold
Dec 27 11:39:31.571495449 (524h10m31.442578514s) [noise=true , address=45.91.116.7:4010     , piece_id=7WLIH23HNUGGBEHPQTF57FSLCPCLDLZU6MKQMC6WM5H2F6VXYXHQ, seg_pos={0 0}, upload=459820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:57836->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:57836->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57836->45.91.116.7:4010: use of closed network connection
Dec 27 11:39:31.571519243 (524h10m31.442602278s) [noise=true , address=45.91.116.7:4010     , piece_id=7WLIH23HNUGGBEHPQTF57FSLCPCLDLZU6MKQMC6WM5H2F6VXYXHQ, seg_pos={0 0}, upload=459820]: Long tail cancellation due to optimal threshold
Dec 27 11:39:31.571621157 (524h10m31.442704202s) [noise=true , address=77.90.31.15:4700     , piece_id=2N7ZK2YFNLJ2REQSAUPHAWWAEL3N7Y4VNFU4YQAERK7KH52PKFHA, seg_pos={0 0}, upload=459820]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF
Dec 27 11:39:31.571629944 (524h10m31.442712979s) [noise=true , address=77.90.31.15:4700     , piece_id=2N7ZK2YFNLJ2REQSAUPHAWWAEL3N7Y4VNFU4YQAERK7KH52PKFHA, seg_pos={0 0}, upload=459820]: Long tail cancellation due to optimal threshold
Dec 27 11:39:31.571648149 (524h10m31.442731184s) [seg_pos={0 0}, upload=459820]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:39:31.571770451 (524h10m31.442853496s) [upload=459820]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:39:31.571775812 (524h10m31.442858847s) [upload=459820]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:39:31.904452356 (524h10m31.775535401s) [upload=459820]: Done uploading segments...
Dec 27 11:39:31.904476762 (524h10m31.775559807s) [upload=459820]: Upload finished. err: <nil>
Duration: 30.002588142s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [......X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X....]
duration: 
6.186773176s
trace: 
https://jaeger.dev.storj.io/trace/85a674a0439b5f2
FAIL (2025-12-27 11:38:21 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:35:41.841238425 (524h6m41.71232146s) [upload=459765]: Starting upload
Dec 27 11:35:41.841285446 (524h6m41.712368491s) [upload=459765]: Done starting upload
Dec 27 11:35:41.841457533 (524h6m41.712540568s) [upload=459765]: Uploading segments...
Dec 27 11:35:41.841513820 (524h6m41.712596865s) [upload=459765]: Got next segment. Inline: false
Dec 27 11:35:41.842253498 (524h6m41.713336533s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:35:41.842309365 (524h6m41.7133924s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:35:42.093392340 (524h6m41.964475375s) [seg_pos={0 0}, upload=459765]: Begin upload segment...
Dec 27 11:35:42.093448658 (524h6m41.964531693s) [seg_pos={0 0}, upload=459765]: Done begin upload segment.
Dec 27 11:35:42.093495477 (524h6m41.964578512s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.093533359 (524h6m41.964616394s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094341098 (524h6m41.965424133s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094658121 (524h6m41.965741156s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094947592 (524h6m41.966030627s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095247203 (524h6m41.966330248s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095524221 (524h6m41.966607256s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095818521 (524h6m41.966901556s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.096129603 (524h6m41.967212638s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097656080 (524h6m41.968739115s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097861100 (524h6m41.968944135s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.100271821 (524h6m41.971354866s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.102305983 (524h6m41.973389019s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.105498964 (524h6m41.976582019s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.108130796 (524h6m41.979213831s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.110435444 (524h6m41.981518479s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:43.817361075 (524h6m43.688444121s) [upload=459765]: Next returned nil segment
Dec 27 11:35:43.817379720 (524h6m43.688462766s) [upload=459765]: Waiting for error group managing segments...
Dec 27 11:35:44.100446118 (524h6m43.971529163s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.114675794 (524h6m43.985758869s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.115095674 (524h6m43.986178709s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116183655 (524h6m43.98726669s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116377414 (524h6m43.987460449s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116605689 (524h6m43.987688724s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.118229791 (524h6m43.989312827s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.119435067 (524h6m43.990518102s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.123509876 (524h6m43.994592911s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.348699054 (524h6m44.219782089s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.353021005 (524h6m44.22410404s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.365289716 (524h6m44.236372751s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496368683 (524h6m44.367451719s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496388611 (524h6m44.367471646s) [seg_pos={0 0}, upload=459765]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:35:44.496562833 (524h6m44.367645868s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496571159 (524h6m44.367654194s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496734399 (524h6m44.367817434s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection
Dec 27 11:35:44.496750170 (524h6m44.367833205s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496789044 (524h6m44.367872079s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496796488 (524h6m44.367879523s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496885157 (524h6m44.367968192s) [seg_pos={0 0}, upload=459765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:35:44.496986230 (524h6m44.368069275s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:35:44.496992171 (524h6m44.368075206s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:35:44.784529301 (524h6m44.655612336s) [upload=459765]: Done uploading segments...
Dec 27 11:35:44.784549729 (524h6m44.655632764s) [upload=459765]: Upload finished. err: <nil>
Duration: 6.186794476s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [.......X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X...]
duration: 
6.106077356s
trace: 
https://jaeger.dev.storj.io/trace/85a674a0439b5f2
FAIL (2025-12-27 11:38:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:35:41.841238425 (524h6m41.71232146s) [upload=459765]: Starting upload
Dec 27 11:35:41.841285446 (524h6m41.712368491s) [upload=459765]: Done starting upload
Dec 27 11:35:41.841457533 (524h6m41.712540568s) [upload=459765]: Uploading segments...
Dec 27 11:35:41.841513820 (524h6m41.712596865s) [upload=459765]: Got next segment. Inline: false
Dec 27 11:35:41.842253498 (524h6m41.713336533s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:35:41.842309365 (524h6m41.7133924s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:35:42.093392340 (524h6m41.964475375s) [seg_pos={0 0}, upload=459765]: Begin upload segment...
Dec 27 11:35:42.093448658 (524h6m41.964531693s) [seg_pos={0 0}, upload=459765]: Done begin upload segment.
Dec 27 11:35:42.093495477 (524h6m41.964578512s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.093533359 (524h6m41.964616394s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094341098 (524h6m41.965424133s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094658121 (524h6m41.965741156s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094947592 (524h6m41.966030627s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095247203 (524h6m41.966330248s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095524221 (524h6m41.966607256s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095818521 (524h6m41.966901556s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.096129603 (524h6m41.967212638s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097656080 (524h6m41.968739115s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097861100 (524h6m41.968944135s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.100271821 (524h6m41.971354866s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.102305983 (524h6m41.973389019s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.105498964 (524h6m41.976582019s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.108130796 (524h6m41.979213831s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.110435444 (524h6m41.981518479s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:43.817361075 (524h6m43.688444121s) [upload=459765]: Next returned nil segment
Dec 27 11:35:43.817379720 (524h6m43.688462766s) [upload=459765]: Waiting for error group managing segments...
Dec 27 11:35:44.100446118 (524h6m43.971529163s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.114675794 (524h6m43.985758869s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.115095674 (524h6m43.986178709s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116183655 (524h6m43.98726669s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116377414 (524h6m43.987460449s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116605689 (524h6m43.987688724s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.118229791 (524h6m43.989312827s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.119435067 (524h6m43.990518102s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.123509876 (524h6m43.994592911s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.348699054 (524h6m44.219782089s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.353021005 (524h6m44.22410404s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.365289716 (524h6m44.236372751s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496368683 (524h6m44.367451719s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496388611 (524h6m44.367471646s) [seg_pos={0 0}, upload=459765]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:35:44.496562833 (524h6m44.367645868s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496571159 (524h6m44.367654194s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496734399 (524h6m44.367817434s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection
Dec 27 11:35:44.496750170 (524h6m44.367833205s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496789044 (524h6m44.367872079s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496796488 (524h6m44.367879523s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496885157 (524h6m44.367968192s) [seg_pos={0 0}, upload=459765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:35:44.496986230 (524h6m44.368069275s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:35:44.496992171 (524h6m44.368075206s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:35:44.784529301 (524h6m44.655612336s) [upload=459765]: Done uploading segments...
Dec 27 11:35:44.784549729 (524h6m44.655632764s) [upload=459765]: Upload finished. err: <nil>
Duration: 6.106156797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
21 out of 60 is failed (threshold=3) [........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X..]
duration: 
6.341239063s
trace: 
https://jaeger.dev.storj.io/trace/85a674a0439b5f2
FAIL (2025-12-27 11:37:22 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:35:41.841238425 (524h6m41.71232146s) [upload=459765]: Starting upload
Dec 27 11:35:41.841285446 (524h6m41.712368491s) [upload=459765]: Done starting upload
Dec 27 11:35:41.841457533 (524h6m41.712540568s) [upload=459765]: Uploading segments...
Dec 27 11:35:41.841513820 (524h6m41.712596865s) [upload=459765]: Got next segment. Inline: false
Dec 27 11:35:41.842253498 (524h6m41.713336533s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:35:41.842309365 (524h6m41.7133924s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:35:42.093392340 (524h6m41.964475375s) [seg_pos={0 0}, upload=459765]: Begin upload segment...
Dec 27 11:35:42.093448658 (524h6m41.964531693s) [seg_pos={0 0}, upload=459765]: Done begin upload segment.
Dec 27 11:35:42.093495477 (524h6m41.964578512s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.093533359 (524h6m41.964616394s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094341098 (524h6m41.965424133s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094658121 (524h6m41.965741156s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094947592 (524h6m41.966030627s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095247203 (524h6m41.966330248s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095524221 (524h6m41.966607256s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095818521 (524h6m41.966901556s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.096129603 (524h6m41.967212638s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097656080 (524h6m41.968739115s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097861100 (524h6m41.968944135s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.100271821 (524h6m41.971354866s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.102305983 (524h6m41.973389019s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.105498964 (524h6m41.976582019s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.108130796 (524h6m41.979213831s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.110435444 (524h6m41.981518479s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:43.817361075 (524h6m43.688444121s) [upload=459765]: Next returned nil segment
Dec 27 11:35:43.817379720 (524h6m43.688462766s) [upload=459765]: Waiting for error group managing segments...
Dec 27 11:35:44.100446118 (524h6m43.971529163s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.114675794 (524h6m43.985758869s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.115095674 (524h6m43.986178709s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116183655 (524h6m43.98726669s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116377414 (524h6m43.987460449s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116605689 (524h6m43.987688724s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.118229791 (524h6m43.989312827s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.119435067 (524h6m43.990518102s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.123509876 (524h6m43.994592911s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.348699054 (524h6m44.219782089s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.353021005 (524h6m44.22410404s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.365289716 (524h6m44.236372751s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496368683 (524h6m44.367451719s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496388611 (524h6m44.367471646s) [seg_pos={0 0}, upload=459765]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:35:44.496562833 (524h6m44.367645868s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496571159 (524h6m44.367654194s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496734399 (524h6m44.367817434s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection
Dec 27 11:35:44.496750170 (524h6m44.367833205s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496789044 (524h6m44.367872079s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496796488 (524h6m44.367879523s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496885157 (524h6m44.367968192s) [seg_pos={0 0}, upload=459765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:35:44.496986230 (524h6m44.368069275s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:35:44.496992171 (524h6m44.368075206s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:35:44.784529301 (524h6m44.655612336s) [upload=459765]: Done uploading segments...
Dec 27 11:35:44.784549729 (524h6m44.655632764s) [upload=459765]: Upload finished. err: <nil>
Duration: 6.341267728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
22 out of 60 is failed (threshold=3) [X........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X.]
duration: 
5.995577175s
trace: 
https://jaeger.dev.storj.io/trace/85a674a0439b5f2
FAIL (2025-12-27 11:36:29 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:35:41.841238425 (524h6m41.71232146s) [upload=459765]: Starting upload
Dec 27 11:35:41.841285446 (524h6m41.712368491s) [upload=459765]: Done starting upload
Dec 27 11:35:41.841457533 (524h6m41.712540568s) [upload=459765]: Uploading segments...
Dec 27 11:35:41.841513820 (524h6m41.712596865s) [upload=459765]: Got next segment. Inline: false
Dec 27 11:35:41.842253498 (524h6m41.713336533s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:35:41.842309365 (524h6m41.7133924s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:35:42.093392340 (524h6m41.964475375s) [seg_pos={0 0}, upload=459765]: Begin upload segment...
Dec 27 11:35:42.093448658 (524h6m41.964531693s) [seg_pos={0 0}, upload=459765]: Done begin upload segment.
Dec 27 11:35:42.093495477 (524h6m41.964578512s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.093533359 (524h6m41.964616394s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094341098 (524h6m41.965424133s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094658121 (524h6m41.965741156s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094947592 (524h6m41.966030627s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095247203 (524h6m41.966330248s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095524221 (524h6m41.966607256s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095818521 (524h6m41.966901556s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.096129603 (524h6m41.967212638s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097656080 (524h6m41.968739115s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097861100 (524h6m41.968944135s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.100271821 (524h6m41.971354866s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.102305983 (524h6m41.973389019s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.105498964 (524h6m41.976582019s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.108130796 (524h6m41.979213831s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.110435444 (524h6m41.981518479s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:43.817361075 (524h6m43.688444121s) [upload=459765]: Next returned nil segment
Dec 27 11:35:43.817379720 (524h6m43.688462766s) [upload=459765]: Waiting for error group managing segments...
Dec 27 11:35:44.100446118 (524h6m43.971529163s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.114675794 (524h6m43.985758869s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.115095674 (524h6m43.986178709s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116183655 (524h6m43.98726669s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116377414 (524h6m43.987460449s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116605689 (524h6m43.987688724s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.118229791 (524h6m43.989312827s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.119435067 (524h6m43.990518102s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.123509876 (524h6m43.994592911s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.348699054 (524h6m44.219782089s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.353021005 (524h6m44.22410404s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.365289716 (524h6m44.236372751s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496368683 (524h6m44.367451719s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496388611 (524h6m44.367471646s) [seg_pos={0 0}, upload=459765]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:35:44.496562833 (524h6m44.367645868s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496571159 (524h6m44.367654194s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496734399 (524h6m44.367817434s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection
Dec 27 11:35:44.496750170 (524h6m44.367833205s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496789044 (524h6m44.367872079s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496796488 (524h6m44.367879523s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496885157 (524h6m44.367968192s) [seg_pos={0 0}, upload=459765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:35:44.496986230 (524h6m44.368069275s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:35:44.496992171 (524h6m44.368075206s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:35:44.784529301 (524h6m44.655612336s) [upload=459765]: Done uploading segments...
Dec 27 11:35:44.784549729 (524h6m44.655632764s) [upload=459765]: Upload finished. err: <nil>
Duration: 5.995602412s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..X]
duration: 
30.004201054s
trace: 
https://jaeger.dev.storj.io/trace/85a674a0439b5f2
FAIL (2025-12-27 11:36:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:35:41.841238425 (524h6m41.71232146s) [upload=459765]: Starting upload
Dec 27 11:35:41.841285446 (524h6m41.712368491s) [upload=459765]: Done starting upload
Dec 27 11:35:41.841457533 (524h6m41.712540568s) [upload=459765]: Uploading segments...
Dec 27 11:35:41.841513820 (524h6m41.712596865s) [upload=459765]: Got next segment. Inline: false
Dec 27 11:35:41.842253498 (524h6m41.713336533s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:35:41.842309365 (524h6m41.7133924s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:35:42.093392340 (524h6m41.964475375s) [seg_pos={0 0}, upload=459765]: Begin upload segment...
Dec 27 11:35:42.093448658 (524h6m41.964531693s) [seg_pos={0 0}, upload=459765]: Done begin upload segment.
Dec 27 11:35:42.093495477 (524h6m41.964578512s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.093533359 (524h6m41.964616394s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094341098 (524h6m41.965424133s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094658121 (524h6m41.965741156s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.094947592 (524h6m41.966030627s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095247203 (524h6m41.966330248s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095524221 (524h6m41.966607256s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.095818521 (524h6m41.966901556s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.096129603 (524h6m41.967212638s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097656080 (524h6m41.968739115s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.097861100 (524h6m41.968944135s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.100271821 (524h6m41.971354866s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.102305983 (524h6m41.973389019s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.105498964 (524h6m41.976582019s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.108130796 (524h6m41.979213831s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:42.110435444 (524h6m41.981518479s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Uploading piece...
Dec 27 11:35:43.817361075 (524h6m43.688444121s) [upload=459765]: Next returned nil segment
Dec 27 11:35:43.817379720 (524h6m43.688462766s) [upload=459765]: Waiting for error group managing segments...
Dec 27 11:35:44.100446118 (524h6m43.971529163s) [noise=true , address=145.79.227.28:4726   , piece_id=EXF4XFMNAANIERXXW4TK7K2LSMDLCG2R6ZZTSO7ALWHOA3IBPH7Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.114675794 (524h6m43.985758869s) [noise=true , address=145.79.227.10:4402   , piece_id=PVY7NKUN5F22PRQDBPKFBFLVGEWK5CHCTRKAVRBWNATIASHV56DQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.115095674 (524h6m43.986178709s) [noise=true , address=145.79.227.29:4942   , piece_id=CODS42NBLREZKCW6OYEGAKRRO2HKE3O3BF2NFOFNR7FZFNKUGY2Q, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116183655 (524h6m43.98726669s) [noise=true , address=145.79.227.13:4802   , piece_id=22HE3F4JYJVBEETQD5Z2U4NAQO5MSFYKNJIIRDMYIXMZM7WGRMTQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116377414 (524h6m43.987460449s) [noise=true , address=145.79.227.42:4123   , piece_id=X3SVEQ4BHUQYNJ3QXG2X2FK7OKZ3OP2JLF6Q2KYQR2LWAAIXHX3A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.116605689 (524h6m43.987688724s) [noise=true , address=145.79.227.25:4617   , piece_id=C7O577KWXKTH5P63NVYYJ2H2TYEOQCBNYWHYWOXUBJQSPW4LEMNA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.118229791 (524h6m43.989312827s) [noise=true , address=145.79.227.14:4837   , piece_id=6CETBVIFIS3IPIRHOGJCB5RYYOIOTGAXWJ2BABLE3WR6OGEWUUPQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.119435067 (524h6m43.990518102s) [noise=true , address=145.79.227.5:4221    , piece_id=F5HHRJE2CEBGKN6ACGNFIZT5ZZOUHIID3B4H4AW5DYEJ4DVZAQZA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.123509876 (524h6m43.994592911s) [noise=true , address=145.79.227.50:4375   , piece_id=IIBO627IOTF4AVWZD5VIF4KTJIQPCCUX6E3NGY6AQQQLTDCRGJOQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.348699054 (524h6m44.219782089s) [noise=true , address=45.91.116.10:4084    , piece_id=UYD6U3JBIZNRF6YGMBLLHYFKTDOLO54DIWMT5IX5VBTF3IFXMI7A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.353021005 (524h6m44.22410404s) [noise=true , address=45.91.116.11:4118    , piece_id=QX7UXQRHRQAASJ33ATWMXANTHHV5DSAYKPFUJEDQOXGXYUQNSM5A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.365289716 (524h6m44.236372751s) [noise=true , address=45.91.116.6:4975     , piece_id=TSIVECOGLOVVQ2A7RGRR3FCU6WROOGYZQTRXDCILASKVX4MMCNCA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496368683 (524h6m44.367451719s) [noise=true , address=45.91.116.14:4190    , piece_id=XFTYA6KM2ZBJMIGLZLL25ADUSY6YWTN36ODTYUKVEDI4W57VJSMQ, seg_pos={0 0}, upload=459765]: Done uploading piece. err: <nil>
Dec 27 11:35:44.496388611 (524h6m44.367471646s) [seg_pos={0 0}, upload=459765]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:35:44.496562833 (524h6m44.367645868s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496571159 (524h6m44.367654194s) [noise=true , address=77.90.31.19:4844     , piece_id=A2ASMPE5A37NONEYHO43BOXLR5N7QPPRCI47TLZU73BZRTWY76RA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496734399 (524h6m44.367817434s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39678->45.91.116.5:4298: use of closed network connection
Dec 27 11:35:44.496750170 (524h6m44.367833205s) [noise=true , address=45.91.116.5:4298     , piece_id=7AXSTZYUHJTL5LWQF7EOHSCNQH6SNVP57G3US7DTLCM7KQQB7N2A, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496789044 (524h6m44.367872079s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2): protocol: expected piece hash; context canceled; EOF
Dec 27 11:35:44.496796488 (524h6m44.367879523s) [noise=true , address=145.79.227.43:4158   , piece_id=D7XHWNMJQKXDVPB2LGNB32AZWBAHKY45ZPXGSUBO4VII5BDSNCPA, seg_pos={0 0}, upload=459765]: Long tail cancellation due to optimal threshold
Dec 27 11:35:44.496885157 (524h6m44.367968192s) [seg_pos={0 0}, upload=459765]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:35:44.496986230 (524h6m44.368069275s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:35:44.496992171 (524h6m44.368075206s) [upload=459765]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:35:44.784529301 (524h6m44.655612336s) [upload=459765]: Done uploading segments...
Dec 27 11:35:44.784549729 (524h6m44.655632764s) [upload=459765]: Upload finished. err: <nil>
Duration: 30.004227815s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
23 out of 60 is failed (threshold=3) [XXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X..]
duration: 
5.984761543s
trace: 
https://jaeger.dev.storj.io/trace/29c40f7f8015d9fa
FAIL (2025-12-27 11:34:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA) context deadline exceeded
======LOGS======
Dec 27 11:32:40.480377406 (524h3m40.351460441s) [upload=459723]: Starting upload
Dec 27 11:32:40.480451907 (524h3m40.351534942s) [upload=459723]: Done starting upload
Dec 27 11:32:40.480496683 (524h3m40.351579728s) [upload=459723]: Uploading segments...
Dec 27 11:32:40.480700931 (524h3m40.351783977s) [upload=459723]: Got next segment. Inline: false
Dec 27 11:32:40.480742401 (524h3m40.351825436s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:32:40.480748222 (524h3m40.351831267s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:32:40.724091800 (524h3m40.595174845s) [seg_pos={0 0}, upload=459723]: Begin upload segment...
Dec 27 11:32:40.724153146 (524h3m40.595236191s) [seg_pos={0 0}, upload=459723]: Done begin upload segment.
Dec 27 11:32:40.724172894 (524h3m40.595255939s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724187992 (524h3m40.595271037s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724251764 (524h3m40.595334799s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724737579 (524h3m40.595820624s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.726681851 (524h3m40.597764886s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727082485 (524h3m40.59816553s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727616030 (524h3m40.598699075s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727887367 (524h3m40.598970402s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728004640 (524h3m40.599087675s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728467541 (524h3m40.599550586s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728661251 (524h3m40.599744296s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728891519 (524h3m40.599974554s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729336827 (524h3m40.600419872s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729343189 (524h3m40.600426224s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729708314 (524h3m40.600791349s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.730053842 (524h3m40.601136887s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:42.354510321 (524h3m42.225593366s) [upload=459723]: Next returned nil segment
Dec 27 11:32:42.354558503 (524h3m42.225641538s) [upload=459723]: Waiting for error group managing segments...
Dec 27 11:32:42.733171566 (524h3m42.604254611s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737054079 (524h3m42.608137124s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737348178 (524h3m42.608431213s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738229938 (524h3m42.609312983s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738260616 (524h3m42.609343662s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.749988208 (524h3m42.621071253s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.752226219 (524h3m42.623309264s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.753552224 (524h3m42.624635269s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.754976406 (524h3m42.626059451s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.755964657 (524h3m42.627047692s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.756150041 (524h3m42.627233076s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.910355444 (524h3m42.781438479s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914588514 (524h3m42.785671549s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914605777 (524h3m42.785688812s) [seg_pos={0 0}, upload=459723]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:32:42.914884498 (524h3m42.785967543s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914895388 (524h3m42.785978423s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.914937348 (524h3m42.786020383s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914944592 (524h3m42.786027627s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915015818 (524h3m42.786098873s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.915023583 (524h3m42.786106628s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915049902 (524h3m42.786132957s) [seg_pos={0 0}, upload=459723]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:32:42.915140144 (524h3m42.786223189s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:32:42.915145795 (524h3m42.78622883s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:32:43.243334202 (524h3m43.114417237s) [upload=459723]: Done uploading segments...
Dec 27 11:32:43.243349801 (524h3m43.114432836s) [upload=459723]: Upload finished. err: <nil>
Duration: 5.984790016s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X.]
duration: 
5.946794047s
trace: 
https://jaeger.dev.storj.io/trace/29c40f7f8015d9fa
FAIL (2025-12-27 11:33:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA) context deadline exceeded
======LOGS======
Dec 27 11:32:40.480377406 (524h3m40.351460441s) [upload=459723]: Starting upload
Dec 27 11:32:40.480451907 (524h3m40.351534942s) [upload=459723]: Done starting upload
Dec 27 11:32:40.480496683 (524h3m40.351579728s) [upload=459723]: Uploading segments...
Dec 27 11:32:40.480700931 (524h3m40.351783977s) [upload=459723]: Got next segment. Inline: false
Dec 27 11:32:40.480742401 (524h3m40.351825436s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:32:40.480748222 (524h3m40.351831267s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:32:40.724091800 (524h3m40.595174845s) [seg_pos={0 0}, upload=459723]: Begin upload segment...
Dec 27 11:32:40.724153146 (524h3m40.595236191s) [seg_pos={0 0}, upload=459723]: Done begin upload segment.
Dec 27 11:32:40.724172894 (524h3m40.595255939s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724187992 (524h3m40.595271037s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724251764 (524h3m40.595334799s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724737579 (524h3m40.595820624s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.726681851 (524h3m40.597764886s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727082485 (524h3m40.59816553s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727616030 (524h3m40.598699075s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727887367 (524h3m40.598970402s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728004640 (524h3m40.599087675s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728467541 (524h3m40.599550586s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728661251 (524h3m40.599744296s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728891519 (524h3m40.599974554s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729336827 (524h3m40.600419872s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729343189 (524h3m40.600426224s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729708314 (524h3m40.600791349s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.730053842 (524h3m40.601136887s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:42.354510321 (524h3m42.225593366s) [upload=459723]: Next returned nil segment
Dec 27 11:32:42.354558503 (524h3m42.225641538s) [upload=459723]: Waiting for error group managing segments...
Dec 27 11:32:42.733171566 (524h3m42.604254611s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737054079 (524h3m42.608137124s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737348178 (524h3m42.608431213s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738229938 (524h3m42.609312983s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738260616 (524h3m42.609343662s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.749988208 (524h3m42.621071253s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.752226219 (524h3m42.623309264s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.753552224 (524h3m42.624635269s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.754976406 (524h3m42.626059451s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.755964657 (524h3m42.627047692s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.756150041 (524h3m42.627233076s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.910355444 (524h3m42.781438479s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914588514 (524h3m42.785671549s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914605777 (524h3m42.785688812s) [seg_pos={0 0}, upload=459723]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:32:42.914884498 (524h3m42.785967543s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914895388 (524h3m42.785978423s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.914937348 (524h3m42.786020383s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914944592 (524h3m42.786027627s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915015818 (524h3m42.786098873s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.915023583 (524h3m42.786106628s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915049902 (524h3m42.786132957s) [seg_pos={0 0}, upload=459723]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:32:42.915140144 (524h3m42.786223189s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:32:42.915145795 (524h3m42.78622883s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:32:43.243334202 (524h3m43.114417237s) [upload=459723]: Done uploading segments...
Dec 27 11:32:43.243349801 (524h3m43.114432836s) [upload=459723]: Upload finished. err: <nil>
Duration: 5.946840655s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...X]
duration: 
30.003255039s
trace: 
https://jaeger.dev.storj.io/trace/29c40f7f8015d9fa
FAIL (2025-12-27 11:33:09 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, Piece ID: OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA) context deadline exceeded
======LOGS======
Dec 27 11:32:40.480377406 (524h3m40.351460441s) [upload=459723]: Starting upload
Dec 27 11:32:40.480451907 (524h3m40.351534942s) [upload=459723]: Done starting upload
Dec 27 11:32:40.480496683 (524h3m40.351579728s) [upload=459723]: Uploading segments...
Dec 27 11:32:40.480700931 (524h3m40.351783977s) [upload=459723]: Got next segment. Inline: false
Dec 27 11:32:40.480742401 (524h3m40.351825436s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:32:40.480748222 (524h3m40.351831267s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:32:40.724091800 (524h3m40.595174845s) [seg_pos={0 0}, upload=459723]: Begin upload segment...
Dec 27 11:32:40.724153146 (524h3m40.595236191s) [seg_pos={0 0}, upload=459723]: Done begin upload segment.
Dec 27 11:32:40.724172894 (524h3m40.595255939s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724187992 (524h3m40.595271037s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724251764 (524h3m40.595334799s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.724737579 (524h3m40.595820624s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.726681851 (524h3m40.597764886s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727082485 (524h3m40.59816553s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727616030 (524h3m40.598699075s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.727887367 (524h3m40.598970402s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728004640 (524h3m40.599087675s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728467541 (524h3m40.599550586s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728661251 (524h3m40.599744296s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.728891519 (524h3m40.599974554s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729336827 (524h3m40.600419872s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729343189 (524h3m40.600426224s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.729708314 (524h3m40.600791349s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:40.730053842 (524h3m40.601136887s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Uploading piece...
Dec 27 11:32:42.354510321 (524h3m42.225593366s) [upload=459723]: Next returned nil segment
Dec 27 11:32:42.354558503 (524h3m42.225641538s) [upload=459723]: Waiting for error group managing segments...
Dec 27 11:32:42.733171566 (524h3m42.604254611s) [noise=true , address=145.79.227.37:4567   , piece_id=5RCWMZYCFV7UODQ4HCHQZR2NATCVX2K7KHDEOZE4W2HF4C5BZCKA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737054079 (524h3m42.608137124s) [noise=true , address=145.79.227.7:4295    , piece_id=344PIYFGCFAIT23RSBIA23RE2UYZHWC2KIKTCZB2X7YRZY2ARYSQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.737348178 (524h3m42.608431213s) [noise=true , address=145.79.227.36:4517   , piece_id=MTEP5ZWNBQD4CW4I4KIW2UR2R2BZ6LVI4NKQ26RRCFHVV7QWEFEA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738229938 (524h3m42.609312983s) [noise=true , address=145.79.227.16:4909   , piece_id=H3IO7LX6OVFYS3UEFH2E2RQYKSSR2764TG3PGBVWOUAUMJOX74YQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.738260616 (524h3m42.609343662s) [noise=true , address=145.79.227.28:4726   , piece_id=TG3WSOJ3CMWN7CB57JOG7QTSDJOKCXIAH5ZDFVTS42IGA3FURXYA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.749988208 (524h3m42.621071253s) [noise=true , address=145.79.227.48:4301   , piece_id=LVABIRAZ6XUIGWRTSUXRLDHQLQZLTDB4XSJK5BC44TO3OVKEG6AQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.752226219 (524h3m42.623309264s) [noise=true , address=145.79.227.17:4972   , piece_id=SNBIK5JCM74UJGXCNABYLKZMNW3XDOZIGWVHG2YPS7OQ2C32WJHQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.753552224 (524h3m42.624635269s) [noise=true , address=145.79.227.50:4374   , piece_id=PB4BBE7HETLKHUADG2U22DKSE6MH66O4NUQAHIYWWK7BJXWX6MBQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.754976406 (524h3m42.626059451s) [noise=true , address=145.79.227.4:4185    , piece_id=PRTYKCOWECMX77V4G5W5JEBGNS26HREBOZTNPSNQBHTLHYJNF4GQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.755964657 (524h3m42.627047692s) [noise=true , address=145.79.227.46:4601   , piece_id=3O7P53MVVE4G426PR4RIXPD7N355UZXZ5BHHJKLKMQIHWKJO7INA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.756150041 (524h3m42.627233076s) [noise=true , address=145.79.227.20:4438   , piece_id=TSJIHXX7M46ZTXEP2H7HSAHSOXC5PNEJKKTAOWKB6HKWC7FOFHWQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.910355444 (524h3m42.781438479s) [noise=true , address=77.90.31.11:4553     , piece_id=OAH2IWWW2ML7767H4KBOYJBQBSEGHEUNVM66XN5XVXHQ4JXWMXTA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914588514 (524h3m42.785671549s) [noise=true , address=77.90.31.10:4518     , piece_id=UHNGMSWMNYM43XV7ESTBRDUEVX4GG7LOCFJBJNY7GN2DU76BX3CA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: <nil>
Dec 27 11:32:42.914605777 (524h3m42.785688812s) [seg_pos={0 0}, upload=459723]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:32:42.914884498 (524h3m42.785967543s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914895388 (524h3m42.785978423s) [noise=true , address=45.91.116.13:4156    , piece_id=57KVHTHXI64NUBB555ENFMLLSBVOXAEMZYFRTLUFKNV2HVAE2ZAA, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.914937348 (524h3m42.786020383s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.914944592 (524h3m42.786027627s) [noise=true , address=145.79.227.11:4729   , piece_id=BI3HSBQOMZEF773JB3GLJTXRNEJDZ2HMG7UQTMNWRBW2WFL76W5Q, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915015818 (524h3m42.786098873s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 27 11:32:42.915023583 (524h3m42.786106628s) [noise=true , address=45.91.116.6:4973     , piece_id=NIXNNER7GQRN5ZHHKSS2A7EMBER3U2LZVQJOVPVT5VX4B3FZWAMQ, seg_pos={0 0}, upload=459723]: Long tail cancellation due to optimal threshold
Dec 27 11:32:42.915049902 (524h3m42.786132957s) [seg_pos={0 0}, upload=459723]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:32:42.915140144 (524h3m42.786223189s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:32:42.915145795 (524h3m42.78622883s) [upload=459723]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:32:43.243334202 (524h3m43.114417237s) [upload=459723]: Done uploading segments...
Dec 27 11:32:43.243349801 (524h3m43.114432836s) [upload=459723]: Upload finished. err: <nil>
Duration: 30.003279866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
24 out of 60 is failed (threshold=3) [X.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X...]
duration: 
5.614659256s
trace: 
https://jaeger.dev.storj.io/trace/3ea87e4b931065fd
FAIL (2025-12-27 11:32:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:30:17.672237812 (524h1m17.543320857s) [upload=459687]: Starting upload
Dec 27 11:30:17.672299720 (524h1m17.543382755s) [upload=459687]: Done starting upload
Dec 27 11:30:17.672353192 (524h1m17.543436227s) [upload=459687]: Uploading segments...
Dec 27 11:30:17.672575496 (524h1m17.543658541s) [upload=459687]: Got next segment. Inline: false
Dec 27 11:30:17.672615552 (524h1m17.543698597s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:30:17.672622354 (524h1m17.543705399s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:30:17.919238914 (524h1m17.79032196s) [seg_pos={0 0}, upload=459687]: Begin upload segment...
Dec 27 11:30:17.919301332 (524h1m17.790384378s) [seg_pos={0 0}, upload=459687]: Done begin upload segment.
Dec 27 11:30:17.919321521 (524h1m17.790404566s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.919335538 (524h1m17.790418583s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.921988771 (524h1m17.793071816s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.922677211 (524h1m17.793760256s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923175480 (524h1m17.794258525s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923698757 (524h1m17.794781802s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925273796 (524h1m17.796356841s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925468856 (524h1m17.796551901s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927347413 (524h1m17.798430459s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927559698 (524h1m17.798642743s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929299150 (524h1m17.800382195s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929441601 (524h1m17.800524646s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931155485 (524h1m17.80223852s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931867050 (524h1m17.802950095s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.934231122 (524h1m17.805314157s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.936141259 (524h1m17.807224294s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:19.639559963 (524h1m19.510642999s) [upload=459687]: Next returned nil segment
Dec 27 11:30:19.639577517 (524h1m19.510660562s) [upload=459687]: Waiting for error group managing segments...
Dec 27 11:30:19.926170859 (524h1m19.797253904s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.930074142 (524h1m19.801157187s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.936256424 (524h1m19.807339459s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941090288 (524h1m19.812173323s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941390280 (524h1m19.812473325s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.951644355 (524h1m19.82272739s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.957430362 (524h1m19.828513407s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.959783022 (524h1m19.830866058s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.962323851 (524h1m19.833406886s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.192531304 (524h1m20.063614349s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.296351495 (524h1m20.16743453s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.311553874 (524h1m20.182636909s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664903020 (524h1m20.535986055s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664924981 (524h1m20.536008026s) [seg_pos={0 0}, upload=459687]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:30:20.665102379 (524h1m20.536185424s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.665127186 (524h1m20.536210231s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.665472764 (524h1m20.536555809s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:34048->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:30:20.665479647 (524h1m20.536562692s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666011961 (524h1m20.537095006s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.666027631 (524h1m20.537110676s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666053190 (524h1m20.537136235s) [seg_pos={0 0}, upload=459687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:30:20.666157809 (524h1m20.537240854s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:30:20.666163450 (524h1m20.537246495s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:30:20.954192935 (524h1m20.825275981s) [upload=459687]: Done uploading segments...
Dec 27 11:30:20.954217062 (524h1m20.825300107s) [upload=459687]: Upload finished. err: <nil>
Duration: 5.614691549s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X..]
duration: 
6.00206885s
trace: 
https://jaeger.dev.storj.io/trace/3ea87e4b931065fd
FAIL (2025-12-27 11:31:13 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:30:17.672237812 (524h1m17.543320857s) [upload=459687]: Starting upload
Dec 27 11:30:17.672299720 (524h1m17.543382755s) [upload=459687]: Done starting upload
Dec 27 11:30:17.672353192 (524h1m17.543436227s) [upload=459687]: Uploading segments...
Dec 27 11:30:17.672575496 (524h1m17.543658541s) [upload=459687]: Got next segment. Inline: false
Dec 27 11:30:17.672615552 (524h1m17.543698597s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:30:17.672622354 (524h1m17.543705399s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:30:17.919238914 (524h1m17.79032196s) [seg_pos={0 0}, upload=459687]: Begin upload segment...
Dec 27 11:30:17.919301332 (524h1m17.790384378s) [seg_pos={0 0}, upload=459687]: Done begin upload segment.
Dec 27 11:30:17.919321521 (524h1m17.790404566s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.919335538 (524h1m17.790418583s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.921988771 (524h1m17.793071816s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.922677211 (524h1m17.793760256s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923175480 (524h1m17.794258525s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923698757 (524h1m17.794781802s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925273796 (524h1m17.796356841s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925468856 (524h1m17.796551901s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927347413 (524h1m17.798430459s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927559698 (524h1m17.798642743s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929299150 (524h1m17.800382195s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929441601 (524h1m17.800524646s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931155485 (524h1m17.80223852s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931867050 (524h1m17.802950095s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.934231122 (524h1m17.805314157s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.936141259 (524h1m17.807224294s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:19.639559963 (524h1m19.510642999s) [upload=459687]: Next returned nil segment
Dec 27 11:30:19.639577517 (524h1m19.510660562s) [upload=459687]: Waiting for error group managing segments...
Dec 27 11:30:19.926170859 (524h1m19.797253904s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.930074142 (524h1m19.801157187s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.936256424 (524h1m19.807339459s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941090288 (524h1m19.812173323s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941390280 (524h1m19.812473325s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.951644355 (524h1m19.82272739s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.957430362 (524h1m19.828513407s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.959783022 (524h1m19.830866058s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.962323851 (524h1m19.833406886s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.192531304 (524h1m20.063614349s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.296351495 (524h1m20.16743453s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.311553874 (524h1m20.182636909s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664903020 (524h1m20.535986055s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664924981 (524h1m20.536008026s) [seg_pos={0 0}, upload=459687]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:30:20.665102379 (524h1m20.536185424s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.665127186 (524h1m20.536210231s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.665472764 (524h1m20.536555809s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:34048->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:30:20.665479647 (524h1m20.536562692s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666011961 (524h1m20.537095006s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.666027631 (524h1m20.537110676s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666053190 (524h1m20.537136235s) [seg_pos={0 0}, upload=459687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:30:20.666157809 (524h1m20.537240854s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:30:20.666163450 (524h1m20.537246495s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:30:20.954192935 (524h1m20.825275981s) [upload=459687]: Done uploading segments...
Dec 27 11:30:20.954217062 (524h1m20.825300107s) [upload=459687]: Upload finished. err: <nil>
Duration: 6.002097204s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X.]
duration: 
6.014519468s
trace: 
https://jaeger.dev.storj.io/trace/3ea87e4b931065fd
FAIL (2025-12-27 11:30:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:30:17.672237812 (524h1m17.543320857s) [upload=459687]: Starting upload
Dec 27 11:30:17.672299720 (524h1m17.543382755s) [upload=459687]: Done starting upload
Dec 27 11:30:17.672353192 (524h1m17.543436227s) [upload=459687]: Uploading segments...
Dec 27 11:30:17.672575496 (524h1m17.543658541s) [upload=459687]: Got next segment. Inline: false
Dec 27 11:30:17.672615552 (524h1m17.543698597s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:30:17.672622354 (524h1m17.543705399s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:30:17.919238914 (524h1m17.79032196s) [seg_pos={0 0}, upload=459687]: Begin upload segment...
Dec 27 11:30:17.919301332 (524h1m17.790384378s) [seg_pos={0 0}, upload=459687]: Done begin upload segment.
Dec 27 11:30:17.919321521 (524h1m17.790404566s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.919335538 (524h1m17.790418583s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.921988771 (524h1m17.793071816s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.922677211 (524h1m17.793760256s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923175480 (524h1m17.794258525s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923698757 (524h1m17.794781802s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925273796 (524h1m17.796356841s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925468856 (524h1m17.796551901s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927347413 (524h1m17.798430459s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927559698 (524h1m17.798642743s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929299150 (524h1m17.800382195s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929441601 (524h1m17.800524646s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931155485 (524h1m17.80223852s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931867050 (524h1m17.802950095s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.934231122 (524h1m17.805314157s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.936141259 (524h1m17.807224294s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:19.639559963 (524h1m19.510642999s) [upload=459687]: Next returned nil segment
Dec 27 11:30:19.639577517 (524h1m19.510660562s) [upload=459687]: Waiting for error group managing segments...
Dec 27 11:30:19.926170859 (524h1m19.797253904s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.930074142 (524h1m19.801157187s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.936256424 (524h1m19.807339459s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941090288 (524h1m19.812173323s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941390280 (524h1m19.812473325s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.951644355 (524h1m19.82272739s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.957430362 (524h1m19.828513407s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.959783022 (524h1m19.830866058s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.962323851 (524h1m19.833406886s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.192531304 (524h1m20.063614349s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.296351495 (524h1m20.16743453s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.311553874 (524h1m20.182636909s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664903020 (524h1m20.535986055s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664924981 (524h1m20.536008026s) [seg_pos={0 0}, upload=459687]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:30:20.665102379 (524h1m20.536185424s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.665127186 (524h1m20.536210231s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.665472764 (524h1m20.536555809s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:34048->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:30:20.665479647 (524h1m20.536562692s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666011961 (524h1m20.537095006s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.666027631 (524h1m20.537110676s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666053190 (524h1m20.537136235s) [seg_pos={0 0}, upload=459687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:30:20.666157809 (524h1m20.537240854s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:30:20.666163450 (524h1m20.537246495s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:30:20.954192935 (524h1m20.825275981s) [upload=459687]: Done uploading segments...
Dec 27 11:30:20.954217062 (524h1m20.825300107s) [upload=459687]: Upload finished. err: <nil>
Duration: 6.014535067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..X]
duration: 
30.003431326s
trace: 
https://jaeger.dev.storj.io/trace/3ea87e4b931065fd
FAIL (2025-12-27 11:30:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:30:17.672237812 (524h1m17.543320857s) [upload=459687]: Starting upload
Dec 27 11:30:17.672299720 (524h1m17.543382755s) [upload=459687]: Done starting upload
Dec 27 11:30:17.672353192 (524h1m17.543436227s) [upload=459687]: Uploading segments...
Dec 27 11:30:17.672575496 (524h1m17.543658541s) [upload=459687]: Got next segment. Inline: false
Dec 27 11:30:17.672615552 (524h1m17.543698597s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:30:17.672622354 (524h1m17.543705399s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:30:17.919238914 (524h1m17.79032196s) [seg_pos={0 0}, upload=459687]: Begin upload segment...
Dec 27 11:30:17.919301332 (524h1m17.790384378s) [seg_pos={0 0}, upload=459687]: Done begin upload segment.
Dec 27 11:30:17.919321521 (524h1m17.790404566s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.919335538 (524h1m17.790418583s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.921988771 (524h1m17.793071816s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.922677211 (524h1m17.793760256s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923175480 (524h1m17.794258525s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.923698757 (524h1m17.794781802s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925273796 (524h1m17.796356841s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.925468856 (524h1m17.796551901s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927347413 (524h1m17.798430459s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.927559698 (524h1m17.798642743s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929299150 (524h1m17.800382195s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.929441601 (524h1m17.800524646s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931155485 (524h1m17.80223852s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.931867050 (524h1m17.802950095s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.934231122 (524h1m17.805314157s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:17.936141259 (524h1m17.807224294s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Uploading piece...
Dec 27 11:30:19.639559963 (524h1m19.510642999s) [upload=459687]: Next returned nil segment
Dec 27 11:30:19.639577517 (524h1m19.510660562s) [upload=459687]: Waiting for error group managing segments...
Dec 27 11:30:19.926170859 (524h1m19.797253904s) [noise=true , address=145.79.227.51:4410   , piece_id=R4AWGBGCZA3CKH5AZWT3NWM5GHBW3V6L44S2ALNZ363JTAINXCEA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.930074142 (524h1m19.801157187s) [noise=true , address=145.79.227.36:4517   , piece_id=TUBPSUFBBZXOH5Y7F3B6LYAGBBR56NEYE644IZTA3ENMOWX6WDJA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.936256424 (524h1m19.807339459s) [noise=true , address=145.79.227.37:4565   , piece_id=PREX7XG42ZTU4YRI6XVUNEFLIWTHCTOFI23UMYKBLKKSEGB447KQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941090288 (524h1m19.812173323s) [noise=true , address=145.79.227.20:4437   , piece_id=WYOIIL6E24JO6552YM6D2LMQLP6L5BNX7BW7IVD6BCJMXH2EEV3Q, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.941390280 (524h1m19.812473325s) [noise=true , address=145.79.227.5:4221    , piece_id=ZCJHIKBO32A4K65FH57PWK52XEDX55QL63ICGRYRNPIJOW4NX2IQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.951644355 (524h1m19.82272739s) [noise=true , address=145.79.227.17:4972   , piece_id=O6XA4YYWX2UT3Q7OWQ7NUFCG2OHPHJCGRQ7ROQGGLHFY4YPZL7QQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.957430362 (524h1m19.828513407s) [noise=true , address=145.79.227.29:4943   , piece_id=OQBED2BUGXKDKMZ52HGH4MV3DUHANW4ZB6JXAQ6EUNKZW6E64MBA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.959783022 (524h1m19.830866058s) [noise=true , address=145.79.227.38:4979   , piece_id=ZB4HT4UDPP42TCLWZWPI4IJNPKGEZQF7PDM4KHOJ6EX6YDEDW6JA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:19.962323851 (524h1m19.833406886s) [noise=true , address=145.79.227.13:4802   , piece_id=HEC67QSHKHSJA2QVY45B45LA6SWRPCKVLAOEHO5D5HGQCYDMLYKQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.192531304 (524h1m20.063614349s) [noise=true , address=45.91.116.6:4975     , piece_id=P3VYBJYGV6Q4DBZKNFCKSWWB2A4PKAKVWXHOBT4ALDXDWZIUWWZQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.296351495 (524h1m20.16743453s) [noise=true , address=45.91.116.5:4299     , piece_id=TL3HYRTKEFXOSI2QJXJHH6PBQSVHVVCDIESKPQIRQEZMC2RE7AGQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.311553874 (524h1m20.182636909s) [noise=true , address=45.91.116.9:4262     , piece_id=NCRVR2KSQVMTAC2HC64P5GV6G52ESJ4VIY2L3D72FWJHT2QC6MHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664903020 (524h1m20.535986055s) [noise=true , address=45.91.116.8:4048     , piece_id=HKP5J6KKSS2R747P2GAZGB2VUUX5QQAL2JHUR33VZALZDWUIHNUA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: <nil>
Dec 27 11:30:20.664924981 (524h1m20.536008026s) [seg_pos={0 0}, upload=459687]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:30:20.665102379 (524h1m20.536185424s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.665127186 (524h1m20.536210231s) [noise=true , address=45.91.116.11:4120    , piece_id=ME26VL5EHGBWWJQIDNFWB6X6RLEFMJS7GM6BB44OV6NUTJXGFJDQ, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.665472764 (524h1m20.536555809s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: read tcp 188.166.225.170:34048->45.91.116.10:4083: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 27 11:30:20.665479647 (524h1m20.536562692s) [noise=true , address=45.91.116.10:4083    , piece_id=SDS7B6FT2W4EE6GWDI2KLWGQVXZEGZX45RJ6FM5ESZFGVDBIZAZA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666011961 (524h1m20.537095006s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF
Dec 27 11:30:20.666027631 (524h1m20.537110676s) [noise=true , address=77.90.31.21:4880     , piece_id=TRJER23XODZWU45QZ2WYL7DDS2M746A5HP3TGHDN5OBNE2TQHHHA, seg_pos={0 0}, upload=459687]: Long tail cancellation due to optimal threshold
Dec 27 11:30:20.666053190 (524h1m20.537136235s) [seg_pos={0 0}, upload=459687]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:30:20.666157809 (524h1m20.537240854s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:30:20.666163450 (524h1m20.537246495s) [upload=459687]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:30:20.954192935 (524h1m20.825275981s) [upload=459687]: Done uploading segments...
Dec 27 11:30:20.954217062 (524h1m20.825300107s) [upload=459687]: Upload finished. err: <nil>
Duration: 30.003457336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [.X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X..]
duration: 
5.697535526s
trace: 
https://jaeger.dev.storj.io/trace/8672f543629964b
FAIL (2025-12-27 11:29:49 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA) context deadline exceeded
======LOGS======
 27 11:27:39.225373904 (523h58m39.096456939s) [upload=459651]: Starting upload
Dec 27 11:27:39.225459026 (523h58m39.096542081s) [upload=459651]: Done starting upload
Dec 27 11:27:39.225641684 (523h58m39.096724729s) [upload=459651]: Uploading segments...
Dec 27 11:27:39.225780358 (523h58m39.096863403s) [upload=459651]: Got next segment. Inline: false
Dec 27 11:27:39.225978806 (523h58m39.097061851s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:27:39.225986190 (523h58m39.097069225s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:27:39.470471501 (523h58m39.341554546s) [seg_pos={0 0}, upload=459651]: Begin upload segment...
Dec 27 11:27:39.470535303 (523h58m39.341618338s) [seg_pos={0 0}, upload=459651]: Done begin upload segment.
Dec 27 11:27:39.470617759 (523h58m39.341700794s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.470638800 (523h58m39.341721835s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471138992 (523h58m39.342222027s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471638824 (523h58m39.342721869s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472129819 (523h58m39.343212854s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472600655 (523h58m39.3436837s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.473040142 (523h58m39.344123177s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.474708760 (523h58m39.345791795s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.475038137 (523h58m39.346121182s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.477105023 (523h58m39.348188058s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.479241691 (523h58m39.350324736s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.480713945 (523h58m39.35179698s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.481554245 (523h58m39.35263729s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.482950694 (523h58m39.354033729s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.483820180 (523h58m39.354903215s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.484378763 (523h58m39.355461798s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:41.195631697 (523h58m41.066714742s) [upload=459651]: Next returned nil segment
Dec 27 11:27:41.195649310 (523h58m41.066732355s) [upload=459651]: Waiting for error group managing segments...
Dec 27 11:27:41.479723735 (523h58m41.35080677s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483476681 (523h58m41.354559726s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483525936 (523h58m41.354608971s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.486140745 (523h58m41.35722379s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.491385140 (523h58m41.362468175s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.503239233 (523h58m41.374322268s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508356327 (523h58m41.379439362s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508668511 (523h58m41.379751546s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.509179284 (523h58m41.380262319s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.511729139 (523h58m41.382812184s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.513883672 (523h58m41.384966707s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.652583589 (523h58m41.523666624s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851675284 (523h58m41.722758319s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851694841 (523h58m41.722777876s) [seg_pos={0 0}, upload=459651]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:27:41.851917645 (523h58m41.72300069s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection
Dec 27 11:27:41.851924338 (523h58m41.723007373s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852110061 (523h58m41.723193096s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852142303 (523h58m41.723225338s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852265296 (523h58m41.723348332s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852271890 (523h58m41.723354925s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852290224 (523h58m41.723373269s) [seg_pos={0 0}, upload=459651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:27:41.852381198 (523h58m41.723464233s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:27:41.852386618 (523h58m41.723469653s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:27:42.145941195 (523h58m42.01702423s) [upload=459651]: Done uploading segments...
Dec 27 11:27:42.145963247 (523h58m42.017046292s) [upload=459651]: Upload finished. err: <nil>
Duration: 5.697561827s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X.]
duration: 
5.804896469s
trace: 
https://jaeger.dev.storj.io/trace/8672f543629964b
FAIL (2025-12-27 11:28:48 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA) context deadline exceeded
======LOGS======
 27 11:27:39.225373904 (523h58m39.096456939s) [upload=459651]: Starting upload
Dec 27 11:27:39.225459026 (523h58m39.096542081s) [upload=459651]: Done starting upload
Dec 27 11:27:39.225641684 (523h58m39.096724729s) [upload=459651]: Uploading segments...
Dec 27 11:27:39.225780358 (523h58m39.096863403s) [upload=459651]: Got next segment. Inline: false
Dec 27 11:27:39.225978806 (523h58m39.097061851s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:27:39.225986190 (523h58m39.097069225s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:27:39.470471501 (523h58m39.341554546s) [seg_pos={0 0}, upload=459651]: Begin upload segment...
Dec 27 11:27:39.470535303 (523h58m39.341618338s) [seg_pos={0 0}, upload=459651]: Done begin upload segment.
Dec 27 11:27:39.470617759 (523h58m39.341700794s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.470638800 (523h58m39.341721835s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471138992 (523h58m39.342222027s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471638824 (523h58m39.342721869s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472129819 (523h58m39.343212854s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472600655 (523h58m39.3436837s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.473040142 (523h58m39.344123177s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.474708760 (523h58m39.345791795s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.475038137 (523h58m39.346121182s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.477105023 (523h58m39.348188058s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.479241691 (523h58m39.350324736s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.480713945 (523h58m39.35179698s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.481554245 (523h58m39.35263729s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.482950694 (523h58m39.354033729s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.483820180 (523h58m39.354903215s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.484378763 (523h58m39.355461798s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:41.195631697 (523h58m41.066714742s) [upload=459651]: Next returned nil segment
Dec 27 11:27:41.195649310 (523h58m41.066732355s) [upload=459651]: Waiting for error group managing segments...
Dec 27 11:27:41.479723735 (523h58m41.35080677s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483476681 (523h58m41.354559726s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483525936 (523h58m41.354608971s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.486140745 (523h58m41.35722379s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.491385140 (523h58m41.362468175s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.503239233 (523h58m41.374322268s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508356327 (523h58m41.379439362s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508668511 (523h58m41.379751546s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.509179284 (523h58m41.380262319s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.511729139 (523h58m41.382812184s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.513883672 (523h58m41.384966707s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.652583589 (523h58m41.523666624s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851675284 (523h58m41.722758319s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851694841 (523h58m41.722777876s) [seg_pos={0 0}, upload=459651]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:27:41.851917645 (523h58m41.72300069s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection
Dec 27 11:27:41.851924338 (523h58m41.723007373s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852110061 (523h58m41.723193096s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852142303 (523h58m41.723225338s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852265296 (523h58m41.723348332s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852271890 (523h58m41.723354925s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852290224 (523h58m41.723373269s) [seg_pos={0 0}, upload=459651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:27:41.852381198 (523h58m41.723464233s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:27:41.852386618 (523h58m41.723469653s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:27:42.145941195 (523h58m42.01702423s) [upload=459651]: Done uploading segments...
Dec 27 11:27:42.145963247 (523h58m42.017046292s) [upload=459651]: Upload finished. err: <nil>
Duration: 5.804913982s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.X]
duration: 
30.004000573s
trace: 
https://jaeger.dev.storj.io/trace/8672f543629964b
FAIL (2025-12-27 11:28:08 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA) context deadline exceeded
======LOGS======
 27 11:27:39.225373904 (523h58m39.096456939s) [upload=459651]: Starting upload
Dec 27 11:27:39.225459026 (523h58m39.096542081s) [upload=459651]: Done starting upload
Dec 27 11:27:39.225641684 (523h58m39.096724729s) [upload=459651]: Uploading segments...
Dec 27 11:27:39.225780358 (523h58m39.096863403s) [upload=459651]: Got next segment. Inline: false
Dec 27 11:27:39.225978806 (523h58m39.097061851s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:27:39.225986190 (523h58m39.097069225s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:27:39.470471501 (523h58m39.341554546s) [seg_pos={0 0}, upload=459651]: Begin upload segment...
Dec 27 11:27:39.470535303 (523h58m39.341618338s) [seg_pos={0 0}, upload=459651]: Done begin upload segment.
Dec 27 11:27:39.470617759 (523h58m39.341700794s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.470638800 (523h58m39.341721835s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471138992 (523h58m39.342222027s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.471638824 (523h58m39.342721869s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472129819 (523h58m39.343212854s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.472600655 (523h58m39.3436837s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.473040142 (523h58m39.344123177s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.474708760 (523h58m39.345791795s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.475038137 (523h58m39.346121182s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.477105023 (523h58m39.348188058s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.479241691 (523h58m39.350324736s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.480713945 (523h58m39.35179698s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.481554245 (523h58m39.35263729s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.482950694 (523h58m39.354033729s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.483820180 (523h58m39.354903215s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:39.484378763 (523h58m39.355461798s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Uploading piece...
Dec 27 11:27:41.195631697 (523h58m41.066714742s) [upload=459651]: Next returned nil segment
Dec 27 11:27:41.195649310 (523h58m41.066732355s) [upload=459651]: Waiting for error group managing segments...
Dec 27 11:27:41.479723735 (523h58m41.35080677s) [noise=true , address=145.79.227.28:4726   , piece_id=R6O5APP7PU6UHOOZBKYTSMGIFOLSZ2ADTISIL722OH4SZLL5PKWQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483476681 (523h58m41.354559726s) [noise=true , address=145.79.227.20:4438   , piece_id=FEVBPU6LJVXF73UHG5T3VVLSDXESU45436E2MMNRVIZHPJJ3OVCA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.483525936 (523h58m41.354608971s) [noise=true , address=145.79.227.29:4942   , piece_id=IBUDWKASL3ABWLFVT7W4VABISY3JJGU6LZUKU42HRK6PWQTDSYCQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.486140745 (523h58m41.35722379s) [noise=true , address=145.79.227.2:4113    , piece_id=T4GCGRL67OQKV46MOVMEUCOFRNPHWS74WLHOCVV24PKG4SGLIGXA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.491385140 (523h58m41.362468175s) [noise=true , address=145.79.227.49:4337   , piece_id=ZELZC4BGHUOFC44JIP7X5TZJPBM3CC4Z6VDY2NQXGU43UHVAWGUA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.503239233 (523h58m41.374322268s) [noise=true , address=145.79.227.46:4602   , piece_id=3Q3IKLE5BYQO66FARWEQGLZJMXWHGTGRTN2WLV4GESUHFNRE5XOA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508356327 (523h58m41.379439362s) [noise=true , address=145.79.227.42:4123   , piece_id=TBU3WZYHOS4FE5JYHFGRGBF5P4UXXVWLMVDWJA6UZIA6U5IPED2Q, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.508668511 (523h58m41.379751546s) [noise=true , address=145.79.227.37:4567   , piece_id=3UUDCEAI5VZGRKOPF3R5DFUBIYR4SSA3W6ZN4RAWMZIC5MRG37FA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.509179284 (523h58m41.380262319s) [noise=true , address=145.79.227.38:4978   , piece_id=JLWE3IMNDNTVIJYRNH4BV5G3T42SXYIZJEYLZ7AKQFLVOIPEL4TQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.511729139 (523h58m41.382812184s) [noise=true , address=145.79.227.14:4837   , piece_id=EQCEMGW3FPQIL2OHRR6XS67TPJG3QYRJEEOBCF3YTVXRQU34PLSA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.513883672 (523h58m41.384966707s) [noise=true , address=145.79.227.23:4546   , piece_id=GOOR2THQYCV4N7GZBRVXOCREDXLROUSL6JGSJDLZ6IWHTJ2FIZGQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.652583589 (523h58m41.523666624s) [noise=true , address=77.90.31.11:4556     , piece_id=EXJAXERIOCSNR32PYXMOVPYDJDMLAZFVVW5HOLOJXCOO7VMYRUHQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851675284 (523h58m41.722758319s) [noise=true , address=45.91.116.14:4190    , piece_id=WLC7UJO7MWD6FAGV4NOBYWNXCCKSXKZ2R7EGO6WQJM4B5GR6K2TA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: <nil>
Dec 27 11:27:41.851694841 (523h58m41.722777876s) [seg_pos={0 0}, upload=459651]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:27:41.851917645 (523h58m41.72300069s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56990->45.91.116.10:4083: use of closed network connection
Dec 27 11:27:41.851924338 (523h58m41.723007373s) [noise=true , address=45.91.116.10:4083    , piece_id=J4SJBUCQIFWEYP63BG7BZPQSH7T6UUMLCQFWBQ4MFKTCJXD474GQ, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852110061 (523h58m41.723193096s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852142303 (523h58m41.723225338s) [noise=true , address=45.91.116.4:4939     , piece_id=XPSAKKHKBTU5NGPYCHK3NNRIPQTOT5DRV6TDJAXASJZHTT55CMIA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852265296 (523h58m41.723348332s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 27 11:27:41.852271890 (523h58m41.723354925s) [noise=true , address=45.91.116.9:4263     , piece_id=LDYRGE2AX7HJSQUFGMEQLX5U4RPB55OQ2BTOPYI6NIPGCWFF25GA, seg_pos={0 0}, upload=459651]: Long tail cancellation due to optimal threshold
Dec 27 11:27:41.852290224 (523h58m41.723373269s) [seg_pos={0 0}, upload=459651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:27:41.852381198 (523h58m41.723464233s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:27:41.852386618 (523h58m41.723469653s) [upload=459651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:27:42.145941195 (523h58m42.01702423s) [upload=459651]: Done uploading segments...
Dec 27 11:27:42.145963247 (523h58m42.017046292s) [upload=459651]: Upload finished. err: <nil>
Duration: 30.004021883s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
25 out of 60 is failed (threshold=3) [.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X.]
duration: 
5.911418197s
trace: 
https://jaeger.dev.storj.io/trace/41a1a192ea70a2ab
FAIL (2025-12-27 11:27:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:26:14.991870153 (523h57m14.862953188s) [upload=459629]: Starting upload
Dec 27 11:26:14.991948872 (523h57m14.863031907s) [upload=459629]: Done starting upload
Dec 27 11:26:14.992071717 (523h57m14.863154752s) [upload=459629]: Uploading segments...
Dec 27 11:26:14.992146669 (523h57m14.863229704s) [upload=459629]: Got next segment. Inline: false
Dec 27 11:26:14.992178159 (523h57m14.863261194s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:26:14.992183569 (523h57m14.863266614s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:26:15.235614142 (523h57m15.106697197s) [seg_pos={0 0}, upload=459629]: Begin upload segment...
Dec 27 11:26:15.235670911 (523h57m15.106753956s) [seg_pos={0 0}, upload=459629]: Done begin upload segment.
Dec 27 11:26:15.235706258 (523h57m15.106789303s) [noise=true , address=145.79.227.29:4942   , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.235778025 (523h57m15.10686107s) [noise=true , address=145.79.227.7:4294    , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.236323093 (523h57m15.107406138s) [noise=true , address=145.79.227.20:4438   , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.236734055 (523h57m15.10781709s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.237171578 (523h57m15.108254613s) [noise=true , address=77.90.31.11:4555     , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.237612979 (523h57m15.108696014s) [noise=true , address=145.79.227.5:4221    , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.238197301 (523h57m15.109280346s) [noise=true , address=145.79.227.17:4971   , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.238637018 (523h57m15.109720053s) [noise=true , address=145.79.227.2:4113    , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.239459916 (523h57m15.110542951s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.240239019 (523h57m15.111322064s) [noise=true , address=45.91.116.6:4976     , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.240618111 (523h57m15.111701146s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.241668441 (523h57m15.112751476s) [noise=true , address=45.91.116.12:4228    , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.242852726 (523h57m15.113935761s) [noise=true , address=145.79.227.48:4303   , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.243884661 (523h57m15.114967696s) [noise=true , address=145.79.227.42:4122   , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.245598725 (523h57m15.11668177s) [noise=true , address=145.79.227.50:4373   , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.245979209 (523h57m15.117062254s) [noise=true , address=145.79.227.28:4726   , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:16.958366904 (523h57m16.829449939s) [upload=459629]: Next returned nil segment
Dec 27 11:26:16.958382122 (523h57m16.829465157s) [upload=459629]: Waiting for error group managing segments...
Dec 27 11:26:17.233391617 (523h57m17.104474662s) [noise=true , address=145.79.227.48:4303   , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.249989252 (523h57m17.121072287s) [noise=true , address=145.79.227.17:4971   , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.253046353 (523h57m17.124129388s) [noise=true , address=145.79.227.29:4942   , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.253447597 (523h57m17.124530642s) [noise=true , address=145.79.227.20:4438   , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.254014657 (523h57m17.125097692s) [noise=true , address=145.79.227.42:4122   , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.254441289 (523h57m17.125524324s) [noise=true , address=145.79.227.28:4726   , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.262470919 (523h57m17.133553954s) [noise=true , address=145.79.227.2:4113    , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.267264416 (523h57m17.138347451s) [noise=true , address=145.79.227.5:4221    , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.269124297 (523h57m17.140207343s) [noise=true , address=145.79.227.7:4294    , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.275073265 (523h57m17.1461563s) [noise=true , address=145.79.227.50:4373   , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.428639562 (523h57m17.299722607s) [noise=true , address=77.90.31.11:4555     , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.483768196 (523h57m17.354851232s) [noise=true , address=45.91.116.6:4976     , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.495914796 (523h57m17.366997831s) [noise=true , address=45.91.116.12:4228    , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.495934413 (523h57m17.367017458s) [seg_pos={0 0}, upload=459629]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:26:17.496159161 (523h57m17.367242206s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496167376 (523h57m17.367250411s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496189268 (523h57m17.367272314s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496210999 (523h57m17.367294034s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496375122 (523h57m17.367458177s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496381043 (523h57m17.367464078s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496403096 (523h57m17.367486131s) [seg_pos={0 0}, upload=459629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:26:17.496465124 (523h57m17.367548169s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:26:17.496476074 (523h57m17.367559109s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:26:17.783203010 (523h57m17.654286075s) [upload=459629]: Done uploading segments...
Dec 27 11:26:17.783221195 (523h57m17.65430424s) [upload=459629]: Upload finished. err: <nil>
Duration: 5.911453795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
26 out of 60 is failed (threshold=3) [X.X..X.XX.XXXX........X.XX.XXXX.X..XX..X......X.X.X...X..X.X]
duration: 
30.003176243s
trace: 
https://jaeger.dev.storj.io/trace/41a1a192ea70a2ab
FAIL (2025-12-27 11:26:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 27 11:26:14.991870153 (523h57m14.862953188s) [upload=459629]: Starting upload
Dec 27 11:26:14.991948872 (523h57m14.863031907s) [upload=459629]: Done starting upload
Dec 27 11:26:14.992071717 (523h57m14.863154752s) [upload=459629]: Uploading segments...
Dec 27 11:26:14.992146669 (523h57m14.863229704s) [upload=459629]: Got next segment. Inline: false
Dec 27 11:26:14.992178159 (523h57m14.863261194s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 27 11:26:14.992183569 (523h57m14.863266614s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 27 11:26:15.235614142 (523h57m15.106697197s) [seg_pos={0 0}, upload=459629]: Begin upload segment...
Dec 27 11:26:15.235670911 (523h57m15.106753956s) [seg_pos={0 0}, upload=459629]: Done begin upload segment.
Dec 27 11:26:15.235706258 (523h57m15.106789303s) [noise=true , address=145.79.227.29:4942   , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.235778025 (523h57m15.10686107s) [noise=true , address=145.79.227.7:4294    , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.236323093 (523h57m15.107406138s) [noise=true , address=145.79.227.20:4438   , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.236734055 (523h57m15.10781709s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.237171578 (523h57m15.108254613s) [noise=true , address=77.90.31.11:4555     , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.237612979 (523h57m15.108696014s) [noise=true , address=145.79.227.5:4221    , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.238197301 (523h57m15.109280346s) [noise=true , address=145.79.227.17:4971   , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.238637018 (523h57m15.109720053s) [noise=true , address=145.79.227.2:4113    , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.239459916 (523h57m15.110542951s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.240239019 (523h57m15.111322064s) [noise=true , address=45.91.116.6:4976     , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.240618111 (523h57m15.111701146s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.241668441 (523h57m15.112751476s) [noise=true , address=45.91.116.12:4228    , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.242852726 (523h57m15.113935761s) [noise=true , address=145.79.227.48:4303   , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.243884661 (523h57m15.114967696s) [noise=true , address=145.79.227.42:4122   , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.245598725 (523h57m15.11668177s) [noise=true , address=145.79.227.50:4373   , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:15.245979209 (523h57m15.117062254s) [noise=true , address=145.79.227.28:4726   , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Uploading piece...
Dec 27 11:26:16.958366904 (523h57m16.829449939s) [upload=459629]: Next returned nil segment
Dec 27 11:26:16.958382122 (523h57m16.829465157s) [upload=459629]: Waiting for error group managing segments...
Dec 27 11:26:17.233391617 (523h57m17.104474662s) [noise=true , address=145.79.227.48:4303   , piece_id=RISWVIJZP3FKBW4JTPI6OCS6RJ7EIJORIVMVWC2JQCCHDVDLEOLQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.249989252 (523h57m17.121072287s) [noise=true , address=145.79.227.17:4971   , piece_id=H45BICB4LRMW6QJLACU2TBANJVS67BNKXPJZRV6EC7OF3LABCKDQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.253046353 (523h57m17.124129388s) [noise=true , address=145.79.227.29:4942   , piece_id=5VPCBV7OVTBAYJ5DIGN7YZCQGGOWAUZMLAKKXGSMEZ2FV4LUHONA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.253447597 (523h57m17.124530642s) [noise=true , address=145.79.227.20:4438   , piece_id=6G5QFBQC7IOLLAZO6NAX4DX3BFZ5N3M5JVTL3K5BPIRTOEIQSL7Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.254014657 (523h57m17.125097692s) [noise=true , address=145.79.227.42:4122   , piece_id=KNFKWWIFMR4OPJ7G53LC7OGFOEAPNSKDUTM6JO53VZGOHFKQHH4A, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.254441289 (523h57m17.125524324s) [noise=true , address=145.79.227.28:4726   , piece_id=UHKEVP22MG2OBXFFC2NGY4IEOTVY6PCK6FYE4PXAVWEQOINXIKQQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.262470919 (523h57m17.133553954s) [noise=true , address=145.79.227.2:4113    , piece_id=6R2UOJX6TWTDEUK4FFRFALT3RUAU2J5RCQ6J37BXP4LFT575EA6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.267264416 (523h57m17.138347451s) [noise=true , address=145.79.227.5:4221    , piece_id=CEHWQXJ6HNVDBD25WHVW2MHVIFSTDTUWMXAKEFWHPEVV2BW7C7YA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.269124297 (523h57m17.140207343s) [noise=true , address=145.79.227.7:4294    , piece_id=RM3L2YM55PYC5B4FRUIGU5TF6IOX7HSIWDEBLL2I7Z2XDEOYNP5Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.275073265 (523h57m17.1461563s) [noise=true , address=145.79.227.50:4373   , piece_id=KRHXXJE5B572UYAOZMWTFJNRIUPI7JI3MFSA7N3AQAW43EITRKXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.428639562 (523h57m17.299722607s) [noise=true , address=77.90.31.11:4555     , piece_id=EYYSCNXCUE7APN5IRSPAS6UAT7DFOWWYCGXTI7V4INS4HRUTMK6Q, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.483768196 (523h57m17.354851232s) [noise=true , address=45.91.116.6:4976     , piece_id=PWV22R54WUKLZF7SOKJGQTEVA7LASCTVRKAJ7VGUWVWYJYYHUKSA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.495914796 (523h57m17.366997831s) [noise=true , address=45.91.116.12:4228    , piece_id=C4SXXUUC44B32DPTRQQE6IIX6LX3L3WOWJ7FM676V5QJXS23HLKQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: <nil>
Dec 27 11:26:17.495934413 (523h57m17.367017458s) [seg_pos={0 0}, upload=459629]: Segment reached optimal threshold of 13 pieces.
Dec 27 11:26:17.496159161 (523h57m17.367242206s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496167376 (523h57m17.367250411s) [noise=true , address=45.91.116.8:4047     , piece_id=LJAL4GBMOZEPR7EPMCPS2KGY4KDGHJG5MTAAWL7EC27UUKWLPMXQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496189268 (523h57m17.367272314s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496210999 (523h57m17.367294034s) [noise=true , address=77.90.31.16:4736     , piece_id=5GGMNYEG6D5HNHHYDMTAXI7W7Q26SD4CR4MK6SBOD723CW6OWFXA, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496375122 (523h57m17.367458177s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Dec 27 11:26:17.496381043 (523h57m17.367464078s) [noise=true , address=45.91.116.11:4119    , piece_id=VOZA5P3GJTKY47UDQUOS5YQAT3R7LUSS3XEDM6IE4VFJKPRKSDTQ, seg_pos={0 0}, upload=459629]: Long tail cancellation due to optimal threshold
Dec 27 11:26:17.496403096 (523h57m17.367486131s) [seg_pos={0 0}, upload=459629]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 27 11:26:17.496465124 (523h57m17.367548169s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 27 11:26:17.496476074 (523h57m17.367559109s) [upload=459629]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 27 11:26:17.783203010 (523h57m17.654286075s) [upload=459629]: Done uploading segments...
Dec 27 11:26:17.783221195 (523h57m17.65430424s) [upload=459629]: Upload finished. err: <nil>
Duration: 30.00320031s core uplink us-select-1