Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 345108 reports (Page 175 of 6903)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [...X...X.............................X...X.X...X..X..X.X..X.]
duration: 
5.949709469s
trace: 
https://jaeger.dev.storj.io/trace/50383e94a2ef611d
FAIL (2025-12-29 00:25:18 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ) context deadline exceeded
======LOGS======
Dec 29 00:24:42.949490598 (560h55m42.820573633s) [upload=491774]: Starting upload
Dec 29 00:24:42.949553929 (560h55m42.820636984s) [upload=491774]: Done starting upload
Dec 29 00:24:42.949587483 (560h55m42.820670518s) [upload=491774]: Uploading segments...
Dec 29 00:24:42.949778466 (560h55m42.820861501s) [upload=491774]: Got next segment. Inline: false
Dec 29 00:24:42.949824414 (560h55m42.820907449s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:24:42.949830736 (560h55m42.820913771s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:24:43.194456640 (560h55m43.065539675s) [seg_pos={0 0}, upload=491774]: Begin upload segment...
Dec 29 00:24:43.194510333 (560h55m43.065593378s) [seg_pos={0 0}, upload=491774]: Done begin upload segment.
Dec 29 00:24:43.194562321 (560h55m43.065645366s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.194574224 (560h55m43.065657259s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195248268 (560h55m43.066331303s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195553659 (560h55m43.066636694s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195571603 (560h55m43.066654648s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196044564 (560h55m43.067127599s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196475574 (560h55m43.067558609s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196829118 (560h55m43.067912153s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197226635 (560h55m43.06830967s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197611127 (560h55m43.068694162s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199252673 (560h55m43.070335718s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199300494 (560h55m43.070383529s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199326343 (560h55m43.070409388s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.208763172 (560h55m43.079846207s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.213399089 (560h55m43.084482135s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.218974406 (560h55m43.090057451s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:44.909291255 (560h55m44.780374291s) [upload=491774]: Next returned nil segment
Dec 29 00:24:44.909311463 (560h55m44.780394498s) [upload=491774]: Waiting for error group managing segments...
Dec 29 00:24:45.200338041 (560h55m45.071421076s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.205843164 (560h55m45.076926199s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.211214101 (560h55m45.082297136s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.220374954 (560h55m45.091457989s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.224941009 (560h55m45.096024054s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.228064747 (560h55m45.099147782s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.232510042 (560h55m45.103593077s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.242384935 (560h55m45.11346797s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.439676608 (560h55m45.310759643s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.480174381 (560h55m45.351257416s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.536506666 (560h55m45.407589701s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.539246563 (560h55m45.410329618s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.549986663 (560h55m45.421069708s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.550009838 (560h55m45.421092883s) [seg_pos={0 0}, upload=491774]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:24:45.550268289 (560h55m45.421351354s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550279451 (560h55m45.421362506s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550341218 (560h55m45.421424263s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550358391 (560h55m45.421441476s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550454474 (560h55m45.42153753s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550462089 (560h55m45.421545134s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550482527 (560h55m45.421565562s) [seg_pos={0 0}, upload=491774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:24:45.550566908 (560h55m45.421649943s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:24:45.550574102 (560h55m45.421657147s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:24:45.847821875 (560h55m45.71890491s) [upload=491774]: Done uploading segments...
Dec 29 00:24:45.847845210 (560h55m45.718928255s) [upload=491774]: Upload finished. err: <nil>
Duration: 5.94972565s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [....X...X.............................X...X.X...X..X..X.X..X]
duration: 
30.003559295s
trace: 
https://jaeger.dev.storj.io/trace/50383e94a2ef611d
FAIL (2025-12-29 00:25:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ) context deadline exceeded
======LOGS======
Dec 29 00:24:42.949490598 (560h55m42.820573633s) [upload=491774]: Starting upload
Dec 29 00:24:42.949553929 (560h55m42.820636984s) [upload=491774]: Done starting upload
Dec 29 00:24:42.949587483 (560h55m42.820670518s) [upload=491774]: Uploading segments...
Dec 29 00:24:42.949778466 (560h55m42.820861501s) [upload=491774]: Got next segment. Inline: false
Dec 29 00:24:42.949824414 (560h55m42.820907449s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:24:42.949830736 (560h55m42.820913771s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:24:43.194456640 (560h55m43.065539675s) [seg_pos={0 0}, upload=491774]: Begin upload segment...
Dec 29 00:24:43.194510333 (560h55m43.065593378s) [seg_pos={0 0}, upload=491774]: Done begin upload segment.
Dec 29 00:24:43.194562321 (560h55m43.065645366s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.194574224 (560h55m43.065657259s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195248268 (560h55m43.066331303s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195553659 (560h55m43.066636694s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.195571603 (560h55m43.066654648s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196044564 (560h55m43.067127599s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196475574 (560h55m43.067558609s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.196829118 (560h55m43.067912153s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197226635 (560h55m43.06830967s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.197611127 (560h55m43.068694162s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199252673 (560h55m43.070335718s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199300494 (560h55m43.070383529s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.199326343 (560h55m43.070409388s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.208763172 (560h55m43.079846207s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.213399089 (560h55m43.084482135s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:43.218974406 (560h55m43.090057451s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Uploading piece...
Dec 29 00:24:44.909291255 (560h55m44.780374291s) [upload=491774]: Next returned nil segment
Dec 29 00:24:44.909311463 (560h55m44.780394498s) [upload=491774]: Waiting for error group managing segments...
Dec 29 00:24:45.200338041 (560h55m45.071421076s) [noise=true , address=145.79.227.7:4293    , piece_id=7PXD46G6X4CXV7LOX7PLJPFOCT4BWHQKQ3LP6PJRUASYKUDNK3XQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.205843164 (560h55m45.076926199s) [noise=true , address=145.79.227.5:4222    , piece_id=NW3TMGBBHGESSBMJ42AC2R7TNBV6VJPNYQNX4LXB4B6XZOQC7MGA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.211214101 (560h55m45.082297136s) [noise=true , address=145.79.227.39:4015   , piece_id=HXS4M7DDYMVC5KFDFILGL4JVV36O22HC6DNOMKFBSRY2HGZTADBQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.220374954 (560h55m45.091457989s) [noise=true , address=145.79.227.51:4411   , piece_id=VS3M6XELAMEA7A23VW53FKB2FHHI3POOHH4RJ42X6HGW7NHILN6Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.224941009 (560h55m45.096024054s) [noise=true , address=145.79.227.47:4266   , piece_id=E3JGDN7P7V2LG2MYY37DK45Q66MH5THDADG4DCS533M543KCWKGQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.228064747 (560h55m45.099147782s) [noise=true , address=145.79.227.32:4836   , piece_id=3K4RGROKYQJRSBKJBVEVSMK7FEUXSKTDYA5QQAKXP6IEMWQX3EWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.232510042 (560h55m45.103593077s) [noise=true , address=145.79.227.14:4840   , piece_id=YKF57JEPRBSKTDHS5X74MDIZO52ZOCAUQNE4VRKPM53ZEZL4ZU2Q, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.242384935 (560h55m45.11346797s) [noise=true , address=145.79.227.15:4876   , piece_id=FW7RTIIEIDUPO7U4VHILNAUXSEQUQSH3ISVCZZVI3F43PHIWMVUA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.439676608 (560h55m45.310759643s) [noise=true , address=45.91.116.14:4191    , piece_id=TQJMEVGFUPOPSJZT5PFN6ZXGATCF2P53EQR4BSTDI6MTV7ZTGWOQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.480174381 (560h55m45.351257416s) [noise=true , address=77.90.31.18:4808     , piece_id=NSOYXKG5OVP3XGU6QF6W7FYJS2MCOWKKUVW4K4BSSAL7IETWL6HQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.536506666 (560h55m45.407589701s) [noise=true , address=45.91.116.11:4120    , piece_id=64MKKHBIPSRYDM6CDSP6V4EQVMO7APNUIRCTGZCDYVHVCSLBYRRQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.539246563 (560h55m45.410329618s) [noise=true , address=45.91.116.13:4156    , piece_id=BSIS2FAXHFTXDBKXSQXWSSFH75BXNFNC22T4Q2AA5S3QVXX3RBDA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.549986663 (560h55m45.421069708s) [noise=true , address=45.91.116.10:4081    , piece_id=HVMUAUQUZ6T6EQVXAP7ZELGZDAV6L62QKMA7WIUERWKRIATRHTLA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: <nil>
Dec 29 00:24:45.550009838 (560h55m45.421092883s) [seg_pos={0 0}, upload=491774]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:24:45.550268289 (560h55m45.421351354s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550279451 (560h55m45.421362506s) [noise=true , address=145.79.227.19:4078   , piece_id=YJGOHLOHSYA7WCZES4UDATNROVAW4LADXPXT63PGAXUQAW47KBWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550341218 (560h55m45.421424263s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550358391 (560h55m45.421441476s) [noise=true , address=77.90.31.14:4661     , piece_id=PKCDMC42ONB6UTP72LISAHFHVN7HFQWCZ2XXP2HZ4465YTQKJRWA, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550454474 (560h55m45.42153753s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:24:45.550462089 (560h55m45.421545134s) [noise=true , address=77.90.31.10:4520     , piece_id=4X5IK72K4OOAANDPW4MYC55IQNUCMI3477YWT24RQEFSRUQ7GKEQ, seg_pos={0 0}, upload=491774]: Long tail cancellation due to optimal threshold
Dec 29 00:24:45.550482527 (560h55m45.421565562s) [seg_pos={0 0}, upload=491774]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:24:45.550566908 (560h55m45.421649943s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:24:45.550574102 (560h55m45.421657147s) [upload=491774]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:24:45.847821875 (560h55m45.71890491s) [upload=491774]: Done uploading segments...
Dec 29 00:24:45.847845210 (560h55m45.718928255s) [upload=491774]: Upload finished. err: <nil>
Duration: 30.003580916s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [.....X...X.............................X...X.X...X..X..X.X..]
duration: 
5.855632157s
trace: 
https://jaeger.dev.storj.io/trace/3892ec88fe9a1bc0
FAIL (2025-12-29 00:23:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
9 00:21:55.476525848 (560h52m55.347608884s) [upload=491734]: Starting upload
Dec 29 00:21:55.476582707 (560h52m55.347665742s) [upload=491734]: Done starting upload
Dec 29 00:21:55.476709448 (560h52m55.347792483s) [upload=491734]: Uploading segments...
Dec 29 00:21:55.476805351 (560h52m55.347888386s) [upload=491734]: Got next segment. Inline: false
Dec 29 00:21:55.476869984 (560h52m55.347953029s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:21:55.476875905 (560h52m55.34795894s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:21:55.728511266 (560h52m55.599594321s) [seg_pos={0 0}, upload=491734]: Begin upload segment...
Dec 29 00:21:55.728564678 (560h52m55.599647743s) [seg_pos={0 0}, upload=491734]: Done begin upload segment.
Dec 29 00:21:55.728645792 (560h52m55.599728837s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.728676661 (560h52m55.599759706s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729479620 (560h52m55.600562655s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729664092 (560h52m55.600747127s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730103177 (560h52m55.601186212s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730193229 (560h52m55.601276264s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730476739 (560h52m55.601559774s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730670417 (560h52m55.601753462s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730826976 (560h52m55.601910011s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731173485 (560h52m55.60225652s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731198714 (560h52m55.602281749s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731595018 (560h52m55.602678063s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731684098 (560h52m55.602767133s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731972106 (560h52m55.603055151s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732165544 (560h52m55.603248579s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732657942 (560h52m55.603741007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:57.449825522 (560h52m57.320908557s) [upload=491734]: Next returned nil segment
Dec 29 00:21:57.449843216 (560h52m57.320926252s) [upload=491734]: Waiting for error group managing segments...
Dec 29 00:21:57.736482759 (560h52m57.607565795s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.738843084 (560h52m57.609926119s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.741299452 (560h52m57.612382487s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.744467254 (560h52m57.615550299s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.745391684 (560h52m57.616474719s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.746580938 (560h52m57.617663983s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.752771807 (560h52m57.623854852s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.756716919 (560h52m57.627799954s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.758525913 (560h52m57.629608948s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.769916122 (560h52m57.640999167s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.910148415 (560h52m57.78123146s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.986458290 (560h52m57.857541325s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081161910 (560h52m57.952244945s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081178260 (560h52m57.952261295s) [seg_pos={0 0}, upload=491734]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:21:58.081330972 (560h52m57.952414007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection
Dec 29 00:21:58.081336221 (560h52m57.952419257s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081405413 (560h52m57.952488448s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081412667 (560h52m57.952495702s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081451160 (560h52m57.952534195s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081458183 (560h52m57.952541228s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081530711 (560h52m57.952613756s) [seg_pos={0 0}, upload=491734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:21:58.081659336 (560h52m57.952742372s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:21:58.081665398 (560h52m57.952748433s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:21:58.378871492 (560h52m58.249954527s) [upload=491734]: Done uploading segments...
Dec 29 00:21:58.378890768 (560h52m58.249973803s) [upload=491734]: Upload finished. err: <nil>
Duration: 5.855658096s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [....................................X...X........X.X........]
duration: 
15.37580498s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:23:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 15.375823175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.....................................X...X........X.X.......]
duration: 
33.281971377s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:23:16 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 33.281997277s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [......................................X...X........X.X......]
duration: 
21.956508657s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:22:38 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 21.956535297s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [......X...X.............................X...X.X...X..X..X.X.]
duration: 
5.941059828s
trace: 
https://jaeger.dev.storj.io/trace/3892ec88fe9a1bc0
FAIL (2025-12-29 00:22:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
9 00:21:55.476525848 (560h52m55.347608884s) [upload=491734]: Starting upload
Dec 29 00:21:55.476582707 (560h52m55.347665742s) [upload=491734]: Done starting upload
Dec 29 00:21:55.476709448 (560h52m55.347792483s) [upload=491734]: Uploading segments...
Dec 29 00:21:55.476805351 (560h52m55.347888386s) [upload=491734]: Got next segment. Inline: false
Dec 29 00:21:55.476869984 (560h52m55.347953029s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:21:55.476875905 (560h52m55.34795894s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:21:55.728511266 (560h52m55.599594321s) [seg_pos={0 0}, upload=491734]: Begin upload segment...
Dec 29 00:21:55.728564678 (560h52m55.599647743s) [seg_pos={0 0}, upload=491734]: Done begin upload segment.
Dec 29 00:21:55.728645792 (560h52m55.599728837s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.728676661 (560h52m55.599759706s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729479620 (560h52m55.600562655s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729664092 (560h52m55.600747127s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730103177 (560h52m55.601186212s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730193229 (560h52m55.601276264s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730476739 (560h52m55.601559774s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730670417 (560h52m55.601753462s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730826976 (560h52m55.601910011s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731173485 (560h52m55.60225652s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731198714 (560h52m55.602281749s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731595018 (560h52m55.602678063s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731684098 (560h52m55.602767133s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731972106 (560h52m55.603055151s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732165544 (560h52m55.603248579s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732657942 (560h52m55.603741007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:57.449825522 (560h52m57.320908557s) [upload=491734]: Next returned nil segment
Dec 29 00:21:57.449843216 (560h52m57.320926252s) [upload=491734]: Waiting for error group managing segments...
Dec 29 00:21:57.736482759 (560h52m57.607565795s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.738843084 (560h52m57.609926119s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.741299452 (560h52m57.612382487s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.744467254 (560h52m57.615550299s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.745391684 (560h52m57.616474719s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.746580938 (560h52m57.617663983s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.752771807 (560h52m57.623854852s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.756716919 (560h52m57.627799954s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.758525913 (560h52m57.629608948s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.769916122 (560h52m57.640999167s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.910148415 (560h52m57.78123146s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.986458290 (560h52m57.857541325s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081161910 (560h52m57.952244945s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081178260 (560h52m57.952261295s) [seg_pos={0 0}, upload=491734]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:21:58.081330972 (560h52m57.952414007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection
Dec 29 00:21:58.081336221 (560h52m57.952419257s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081405413 (560h52m57.952488448s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081412667 (560h52m57.952495702s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081451160 (560h52m57.952534195s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081458183 (560h52m57.952541228s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081530711 (560h52m57.952613756s) [seg_pos={0 0}, upload=491734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:21:58.081659336 (560h52m57.952742372s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:21:58.081665398 (560h52m57.952748433s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:21:58.378871492 (560h52m58.249954527s) [upload=491734]: Done uploading segments...
Dec 29 00:21:58.378890768 (560h52m58.249973803s) [upload=491734]: Upload finished. err: <nil>
Duration: 5.941085017s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
10 out of 60 is failed (threshold=3) [X......X...X.............................X...X.X...X..X..X.X]
duration: 
30.002582379s
trace: 
https://jaeger.dev.storj.io/trace/3892ec88fe9a1bc0
FAIL (2025-12-29 00:22:24 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
9 00:21:55.476525848 (560h52m55.347608884s) [upload=491734]: Starting upload
Dec 29 00:21:55.476582707 (560h52m55.347665742s) [upload=491734]: Done starting upload
Dec 29 00:21:55.476709448 (560h52m55.347792483s) [upload=491734]: Uploading segments...
Dec 29 00:21:55.476805351 (560h52m55.347888386s) [upload=491734]: Got next segment. Inline: false
Dec 29 00:21:55.476869984 (560h52m55.347953029s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:21:55.476875905 (560h52m55.34795894s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:21:55.728511266 (560h52m55.599594321s) [seg_pos={0 0}, upload=491734]: Begin upload segment...
Dec 29 00:21:55.728564678 (560h52m55.599647743s) [seg_pos={0 0}, upload=491734]: Done begin upload segment.
Dec 29 00:21:55.728645792 (560h52m55.599728837s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.728676661 (560h52m55.599759706s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729479620 (560h52m55.600562655s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.729664092 (560h52m55.600747127s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730103177 (560h52m55.601186212s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730193229 (560h52m55.601276264s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730476739 (560h52m55.601559774s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730670417 (560h52m55.601753462s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.730826976 (560h52m55.601910011s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731173485 (560h52m55.60225652s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731198714 (560h52m55.602281749s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731595018 (560h52m55.602678063s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731684098 (560h52m55.602767133s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.731972106 (560h52m55.603055151s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732165544 (560h52m55.603248579s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:55.732657942 (560h52m55.603741007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Uploading piece...
Dec 29 00:21:57.449825522 (560h52m57.320908557s) [upload=491734]: Next returned nil segment
Dec 29 00:21:57.449843216 (560h52m57.320926252s) [upload=491734]: Waiting for error group managing segments...
Dec 29 00:21:57.736482759 (560h52m57.607565795s) [noise=true , address=145.79.227.41:4087   , piece_id=6JCGMDDBX4A7EPTG4UDWBKF6GZ7AHDXYFRV726C6QB2GBFM2VGIA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.738843084 (560h52m57.609926119s) [noise=true , address=145.79.227.16:4909   , piece_id=DD7WHLFOST463YXHHC3SMLFUR27MN77ICG2GZXNTC5HKMFSKE7FQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.741299452 (560h52m57.612382487s) [noise=true , address=145.79.227.48:4302   , piece_id=SRIYQT35PHMVRAJKFDXMM4W7NEG5TWW5BYAEX3KA4H5SGD24GSOA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.744467254 (560h52m57.615550299s) [noise=true , address=145.79.227.11:4732   , piece_id=VOQXYRI2YCXG7GBQVE4QJWIJAIOHWYGM4TYUHIXAPFKU65AVIOVA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.745391684 (560h52m57.616474719s) [noise=true , address=145.79.227.26:4653   , piece_id=HEPX5SCOB7NJZTGIB67DMFAPSAIKESCMZNTOE3GLJSAMTCPJ7KEA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.746580938 (560h52m57.617663983s) [noise=true , address=145.79.227.36:4518   , piece_id=HUYXPTGUJ6ZIUDEZETJUEFIXO4PU7KZMJIIZNR5XMYRCKIFOXRRQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.752771807 (560h52m57.623854852s) [noise=true , address=145.79.227.4:4187    , piece_id=ER2NFGHAETZMQWIA6IH75VQ7JIDI4U6ZCDDEWW4TJ54MHV4D5KSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.756716919 (560h52m57.627799954s) [noise=true , address=145.79.227.24:4582   , piece_id=LBV67LNBO3YV62PZQFCFQOCRUGOX7CIMBVBPIDFY5Q4EJTLCEYBQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.758525913 (560h52m57.629608948s) [noise=true , address=145.79.227.45:4230   , piece_id=MGPG2LDHNZ6DVGXMM7VO72KHEDUVFUFDEAUQB7CWYG2HNUOZXA4A, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.769916122 (560h52m57.640999167s) [noise=true , address=145.79.227.46:4602   , piece_id=RC2CWUF6PO25MW3Y2RHSVNNP3TJBJKUNZDLIIKVQBBBP4HETYGGQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.910148415 (560h52m57.78123146s) [noise=true , address=77.90.31.18:4808     , piece_id=HNBXTO4YOZOHMGUFFAYUMUTRNNRYWJ2BZDG7RPJKUD7FCZVL7SUA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:57.986458290 (560h52m57.857541325s) [noise=true , address=45.91.116.8:4045     , piece_id=P5ONQ7ZKBT6JKFHL3MSHLLK3BHPR2PM43NARCHO2LJI27J5LU4ZQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081161910 (560h52m57.952244945s) [noise=true , address=145.79.227.37:4566   , piece_id=RUUACKYA4CCIVQFUHHH2HIX2BCWDHMKIEXXEBB5U5ZRXZ3K3DYKQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: <nil>
Dec 29 00:21:58.081178260 (560h52m57.952261295s) [seg_pos={0 0}, upload=491734]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:21:58.081330972 (560h52m57.952414007s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54324->45.91.116.9:4261: use of closed network connection
Dec 29 00:21:58.081336221 (560h52m57.952419257s) [noise=true , address=45.91.116.9:4261     , piece_id=CUAWJPKCGEU54YRRYM2AYLBIAQHWG3PYMRQX6BZC2NYNL4T6NGYA, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081405413 (560h52m57.952488448s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081412667 (560h52m57.952495702s) [noise=true , address=77.90.31.13:4626     , piece_id=GBV7F5AH23IZAFOT3BJVPQANGSUDJHRTZRN2J6EWBLA7EDAZEVUQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081451160 (560h52m57.952534195s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF
Dec 29 00:21:58.081458183 (560h52m57.952541228s) [noise=true , address=45.91.116.5:4297     , piece_id=NR24JUVVXBN2HIWCVLTSPGWNVRDJF2P4RBR4NETJHP4FLQR3URSQ, seg_pos={0 0}, upload=491734]: Long tail cancellation due to optimal threshold
Dec 29 00:21:58.081530711 (560h52m57.952613756s) [seg_pos={0 0}, upload=491734]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:21:58.081659336 (560h52m57.952742372s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:21:58.081665398 (560h52m57.952748433s) [upload=491734]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:21:58.378871492 (560h52m58.249954527s) [upload=491734]: Done uploading segments...
Dec 29 00:21:58.378890768 (560h52m58.249973803s) [upload=491734]: Upload finished. err: <nil>
Duration: 30.002604141s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.......................................X...X........X.X.....]
duration: 
24.69891432s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:21:41 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 24.698968884s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [.X......X...X.............................X...X.X...X..X..X.]
duration: 
5.886556173s
trace: 
https://jaeger.dev.storj.io/trace/6d4bba58858cd4b8
FAIL (2025-12-29 00:21:37 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA) context deadline exceeded
======LOGS======
0:20:26.660574014 (560h51m26.531657059s) [upload=491715]: Starting upload
Dec 29 00:20:26.660639498 (560h51m26.531722543s) [upload=491715]: Done starting upload
Dec 29 00:20:26.660755158 (560h51m26.531838193s) [upload=491715]: Uploading segments...
Dec 29 00:20:26.660961501 (560h51m26.532044536s) [upload=491715]: Got next segment. Inline: false
Dec 29 00:20:26.660994995 (560h51m26.53207804s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:20:26.661000005 (560h51m26.53208304s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:20:26.901181324 (560h51m26.772264369s) [seg_pos={0 0}, upload=491715]: Begin upload segment...
Dec 29 00:20:26.901229014 (560h51m26.772312049s) [seg_pos={0 0}, upload=491715]: Done begin upload segment.
Dec 29 00:20:26.901270473 (560h51m26.772353508s) [noise=true , address=45.91.116.8:4045     , piece_id=ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.901314006 (560h51m26.772397042s) [noise=true , address=145.79.227.15:4874   , piece_id=6DCDTJCSLM6DCJRRU34HS2JUVSUEZXMSMHIR3SOP6M4C4EI3QR4Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.901867499 (560h51m26.772950546s) [noise=true , address=145.79.227.24:4582   , piece_id=HC7HUOPCEDO4FWSV55ZPGSW7NPRT4KCWZ4TZ3T5ARRN3EKM75S3A, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902211154 (560h51m26.773294189s) [noise=true , address=45.91.116.7:4012     , piece_id=MIPHXTUKOVZBE5SW6O5PI2AIHPXR4HJ35RAU2NCBPIKL62HFNZWQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902550240 (560h51m26.773633285s) [noise=true , address=77.90.31.19:4841     , piece_id=Z4FTBGVJZGX6YCPBP7HVHEGZFUJMZYF5KODYKIYA2YCJYXMYT7KQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902988685 (560h51m26.77407172s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903320767 (560h51m26.774403802s) [noise=true , address=145.79.227.38:4977   , piece_id=J5ZQMZEWKBLH35UXCUC3P7IWSFSW5GRRUJ672VOBBR5MBVJF6OBA, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903608916 (560h51m26.774691962s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903921622 (560h51m26.775004657s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.905070530 (560h51m26.776153565s) [noise=true , address=145.79.227.13:4804   , piece_id=YCSPBECIR2DUFHKARRT572SNISLJER354NM7WLY4PQSKCJ2EQI7Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.906469934 (560h51m26.777552969s) [noise=true , address=77.90.31.16:4735     , piece_id=MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.907995299 (560h51m26.779078334s) [noise=true , address=145.79.227.18:4006   , piece_id=65NCB5NFVFTZLCRM7MOBNFEUKLIY7IVON7KIXIP3RMQD5443M66Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.909925915 (560h51m26.78100896s) [noise=true , address=145.79.227.45:4231   , piece_id=YL6I5BMBJ5O5VVLS2HSVCHB4CEP3X6D6F23G7JFFOCO5VQDO526A, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.911846652 (560h51m26.782929687s) [noise=true , address=145.79.227.19:4077   , piece_id=FMNNU64BCTV76ZTSOORGV3IRUA26RBC5WUKHHNFTZLDJ6VJZ56OQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.913735869 (560h51m26.784818904s) [noise=true , address=145.79.227.6:4258    , piece_id=T5E2VG6LGVIDHMUNIWSVHO6HTHCMHK5EWY3YCKELPNPQCKXYSFVQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.914263164 (560h51m26.785346199s) [noise=true , address=145.79.227.48:4301   , piece_id=DWZSP7OCN26BDNVI6ATIJF3KIY2OQ6OURAUJC66IACUGZVOBDGBQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:28.619001387 (560h51m28.490084432s) [upload=491715]: Next returned nil segment
Dec 29 00:20:28.619018310 (560h51m28.490101355s) [upload=491715]: Waiting for error group managing segments...
Dec 29 00:20:28.908026943 (560h51m28.779109978s) [noise=true , address=145.79.227.24:4582   , piece_id=HC7HUOPCEDO4FWSV55ZPGSW7NPRT4KCWZ4TZ3T5ARRN3EKM75S3A, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.909267766 (560h51m28.780350801s) [noise=true , address=145.79.227.38:4977   , piece_id=J5ZQMZEWKBLH35UXCUC3P7IWSFSW5GRRUJ672VOBBR5MBVJF6OBA, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.921607262 (560h51m28.792690297s) [noise=true , address=145.79.227.19:4077   , piece_id=FMNNU64BCTV76ZTSOORGV3IRUA26RBC5WUKHHNFTZLDJ6VJZ56OQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.925480838 (560h51m28.796563883s) [noise=true , address=145.79.227.15:4874   , piece_id=6DCDTJCSLM6DCJRRU34HS2JUVSUEZXMSMHIR3SOP6M4C4EI3QR4Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.931164590 (560h51m28.802247626s) [noise=true , address=145.79.227.18:4006   , piece_id=65NCB5NFVFTZLCRM7MOBNFEUKLIY7IVON7KIXIP3RMQD5443M66Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.931854344 (560h51m28.802937379s) [noise=true , address=145.79.227.13:4804   , piece_id=YCSPBECIR2DUFHKARRT572SNISLJER354NM7WLY4PQSKCJ2EQI7Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.936009947 (560h51m28.807092982s) [noise=true , address=145.79.227.45:4231   , piece_id=YL6I5BMBJ5O5VVLS2HSVCHB4CEP3X6D6F23G7JFFOCO5VQDO526A, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.944668383 (560h51m28.815751418s) [noise=true , address=145.79.227.48:4301   , piece_id=DWZSP7OCN26BDNVI6ATIJF3KIY2OQ6OURAUJC66IACUGZVOBDGBQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.980320279 (560h51m28.851403314s) [noise=true , address=145.79.227.6:4258    , piece_id=T5E2VG6LGVIDHMUNIWSVHO6HTHCMHK5EWY3YCKELPNPQCKXYSFVQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.167074186 (560h51m29.038157231s) [noise=true , address=77.90.31.16:4735     , piece_id=MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.184947168 (560h51m29.056030203s) [noise=true , address=77.90.31.19:4841     , piece_id=Z4FTBGVJZGX6YCPBP7HVHEGZFUJMZYF5KODYKIYA2YCJYXMYT7KQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.279518235 (560h51m29.15060128s) [noise=true , address=45.91.116.7:4012     , piece_id=MIPHXTUKOVZBE5SW6O5PI2AIHPXR4HJ35RAU2NCBPIKL62HFNZWQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.907678440 (560h51m29.778761475s) [noise=true , address=45.91.116.8:4045     , piece_id=ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.907703498 (560h51m29.778786543s) [seg_pos={0 0}, upload=491715]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:20:29.907933235 (560h51m29.77901627s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:20:29.907941121 (560h51m29.779024156s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.907996496 (560h51m29.779079531s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection
Dec 29 00:20:29.908011816 (560h51m29.779094861s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.908222276 (560h51m29.779305311s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 29 00:20:29.908229380 (560h51m29.779312415s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.908248516 (560h51m29.779331551s) [seg_pos={0 0}, upload=491715]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:20:29.908320413 (560h51m29.779403448s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:20:29.908325182 (560h51m29.779408217s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:20:30.198680680 (560h51m30.069763715s) [upload=491715]: Done uploading segments...
Dec 29 00:20:30.198705648 (560h51m30.069788683s) [upload=491715]: Upload finished. err: <nil>
Duration: 5.886586602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
9 out of 60 is failed (threshold=3) [..X......X...X.............................X...X.X...X..X..X]
duration: 
30.003507568s
trace: 
https://jaeger.dev.storj.io/trace/6d4bba58858cd4b8
FAIL (2025-12-29 00:20:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ) context deadline exceeded; piecestore: (Node ID: 14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h, Piece ID: MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA) context deadline exceeded
======LOGS======
0:20:26.660574014 (560h51m26.531657059s) [upload=491715]: Starting upload
Dec 29 00:20:26.660639498 (560h51m26.531722543s) [upload=491715]: Done starting upload
Dec 29 00:20:26.660755158 (560h51m26.531838193s) [upload=491715]: Uploading segments...
Dec 29 00:20:26.660961501 (560h51m26.532044536s) [upload=491715]: Got next segment. Inline: false
Dec 29 00:20:26.660994995 (560h51m26.53207804s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:20:26.661000005 (560h51m26.53208304s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:20:26.901181324 (560h51m26.772264369s) [seg_pos={0 0}, upload=491715]: Begin upload segment...
Dec 29 00:20:26.901229014 (560h51m26.772312049s) [seg_pos={0 0}, upload=491715]: Done begin upload segment.
Dec 29 00:20:26.901270473 (560h51m26.772353508s) [noise=true , address=45.91.116.8:4045     , piece_id=ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.901314006 (560h51m26.772397042s) [noise=true , address=145.79.227.15:4874   , piece_id=6DCDTJCSLM6DCJRRU34HS2JUVSUEZXMSMHIR3SOP6M4C4EI3QR4Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.901867499 (560h51m26.772950546s) [noise=true , address=145.79.227.24:4582   , piece_id=HC7HUOPCEDO4FWSV55ZPGSW7NPRT4KCWZ4TZ3T5ARRN3EKM75S3A, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902211154 (560h51m26.773294189s) [noise=true , address=45.91.116.7:4012     , piece_id=MIPHXTUKOVZBE5SW6O5PI2AIHPXR4HJ35RAU2NCBPIKL62HFNZWQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902550240 (560h51m26.773633285s) [noise=true , address=77.90.31.19:4841     , piece_id=Z4FTBGVJZGX6YCPBP7HVHEGZFUJMZYF5KODYKIYA2YCJYXMYT7KQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.902988685 (560h51m26.77407172s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903320767 (560h51m26.774403802s) [noise=true , address=145.79.227.38:4977   , piece_id=J5ZQMZEWKBLH35UXCUC3P7IWSFSW5GRRUJ672VOBBR5MBVJF6OBA, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903608916 (560h51m26.774691962s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.903921622 (560h51m26.775004657s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.905070530 (560h51m26.776153565s) [noise=true , address=145.79.227.13:4804   , piece_id=YCSPBECIR2DUFHKARRT572SNISLJER354NM7WLY4PQSKCJ2EQI7Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.906469934 (560h51m26.777552969s) [noise=true , address=77.90.31.16:4735     , piece_id=MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.907995299 (560h51m26.779078334s) [noise=true , address=145.79.227.18:4006   , piece_id=65NCB5NFVFTZLCRM7MOBNFEUKLIY7IVON7KIXIP3RMQD5443M66Q, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.909925915 (560h51m26.78100896s) [noise=true , address=145.79.227.45:4231   , piece_id=YL6I5BMBJ5O5VVLS2HSVCHB4CEP3X6D6F23G7JFFOCO5VQDO526A, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.911846652 (560h51m26.782929687s) [noise=true , address=145.79.227.19:4077   , piece_id=FMNNU64BCTV76ZTSOORGV3IRUA26RBC5WUKHHNFTZLDJ6VJZ56OQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.913735869 (560h51m26.784818904s) [noise=true , address=145.79.227.6:4258    , piece_id=T5E2VG6LGVIDHMUNIWSVHO6HTHCMHK5EWY3YCKELPNPQCKXYSFVQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:26.914263164 (560h51m26.785346199s) [noise=true , address=145.79.227.48:4301   , piece_id=DWZSP7OCN26BDNVI6ATIJF3KIY2OQ6OURAUJC66IACUGZVOBDGBQ, seg_pos={0 0}, upload=491715]: Uploading piece...
Dec 29 00:20:28.619001387 (560h51m28.490084432s) [upload=491715]: Next returned nil segment
Dec 29 00:20:28.619018310 (560h51m28.490101355s) [upload=491715]: Waiting for error group managing segments...
Dec 29 00:20:28.908026943 (560h51m28.779109978s) [noise=true , address=145.79.227.24:4582   , piece_id=HC7HUOPCEDO4FWSV55ZPGSW7NPRT4KCWZ4TZ3T5ARRN3EKM75S3A, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.909267766 (560h51m28.780350801s) [noise=true , address=145.79.227.38:4977   , piece_id=J5ZQMZEWKBLH35UXCUC3P7IWSFSW5GRRUJ672VOBBR5MBVJF6OBA, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.921607262 (560h51m28.792690297s) [noise=true , address=145.79.227.19:4077   , piece_id=FMNNU64BCTV76ZTSOORGV3IRUA26RBC5WUKHHNFTZLDJ6VJZ56OQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.925480838 (560h51m28.796563883s) [noise=true , address=145.79.227.15:4874   , piece_id=6DCDTJCSLM6DCJRRU34HS2JUVSUEZXMSMHIR3SOP6M4C4EI3QR4Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.931164590 (560h51m28.802247626s) [noise=true , address=145.79.227.18:4006   , piece_id=65NCB5NFVFTZLCRM7MOBNFEUKLIY7IVON7KIXIP3RMQD5443M66Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.931854344 (560h51m28.802937379s) [noise=true , address=145.79.227.13:4804   , piece_id=YCSPBECIR2DUFHKARRT572SNISLJER354NM7WLY4PQSKCJ2EQI7Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.936009947 (560h51m28.807092982s) [noise=true , address=145.79.227.45:4231   , piece_id=YL6I5BMBJ5O5VVLS2HSVCHB4CEP3X6D6F23G7JFFOCO5VQDO526A, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.944668383 (560h51m28.815751418s) [noise=true , address=145.79.227.48:4301   , piece_id=DWZSP7OCN26BDNVI6ATIJF3KIY2OQ6OURAUJC66IACUGZVOBDGBQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:28.980320279 (560h51m28.851403314s) [noise=true , address=145.79.227.6:4258    , piece_id=T5E2VG6LGVIDHMUNIWSVHO6HTHCMHK5EWY3YCKELPNPQCKXYSFVQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.167074186 (560h51m29.038157231s) [noise=true , address=77.90.31.16:4735     , piece_id=MEH3BCTLGM3GZ4J7FDRW3IDWDXPMCO57I3WQLFZ7JGLUTBQENBOA, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.184947168 (560h51m29.056030203s) [noise=true , address=77.90.31.19:4841     , piece_id=Z4FTBGVJZGX6YCPBP7HVHEGZFUJMZYF5KODYKIYA2YCJYXMYT7KQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.279518235 (560h51m29.15060128s) [noise=true , address=45.91.116.7:4012     , piece_id=MIPHXTUKOVZBE5SW6O5PI2AIHPXR4HJ35RAU2NCBPIKL62HFNZWQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.907678440 (560h51m29.778761475s) [noise=true , address=45.91.116.8:4045     , piece_id=ILFV5QMOCPHNWRTWBUX5ZR4PGRPXNTNJNSANZXJV5NPAGUBUODFQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: <nil>
Dec 29 00:20:29.907703498 (560h51m29.778786543s) [seg_pos={0 0}, upload=491715]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:20:29.907933235 (560h51m29.77901627s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:20:29.907941121 (560h51m29.779024156s) [noise=true , address=45.91.116.5:4300     , piece_id=SM4355UUWQAROYLQVSZJ5ANHNWAZVB2ELSA2X7NWE23RQHVY4T4Q, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.907996496 (560h51m29.779079531s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn): protocol: write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection; write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48600->145.79.227.16:4910: use of closed network connection
Dec 29 00:20:29.908011816 (560h51m29.779094861s) [noise=true , address=145.79.227.16:4910   , piece_id=J6I6WVRWDUNVMTX5LXW5MSK3E4SBDNRT5Q572S2ZCXXZXSBG52BQ, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.908222276 (560h51m29.779305311s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 29 00:20:29.908229380 (560h51m29.779312415s) [noise=true , address=145.79.227.49:4339   , piece_id=7CXRBG6C3YO4ZI4J2CWS4E7KPXAGBYGGWQHCAXZMJJQ4FRXTKXYQ, seg_pos={0 0}, upload=491715]: Long tail cancellation due to optimal threshold
Dec 29 00:20:29.908248516 (560h51m29.779331551s) [seg_pos={0 0}, upload=491715]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:20:29.908320413 (560h51m29.779403448s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:20:29.908325182 (560h51m29.779408217s) [upload=491715]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:20:30.198680680 (560h51m30.069763715s) [upload=491715]: Done uploading segments...
Dec 29 00:20:30.198705648 (560h51m30.069788683s) [upload=491715]: Upload finished. err: <nil>
Duration: 30.003527085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [........................................X...X........X.X....]
duration: 
15.797445818s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:20:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 15.79747331s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [...X......X...X.............................X...X.X...X..X..]
duration: 
6.157583331s
trace: 
https://jaeger.dev.storj.io/trace/a8ef1a14dfa7309
FAIL (2025-12-29 00:20:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ) context deadline exceeded
======LOGS======
Dec 29 00:18:59.652723023 (560h49m59.523806058s) [upload=491693]: Starting upload
Dec 29 00:18:59.652774981 (560h49m59.523858017s) [upload=491693]: Done starting upload
Dec 29 00:18:59.652944194 (560h49m59.524027229s) [upload=491693]: Uploading segments...
Dec 29 00:18:59.652980803 (560h49m59.524063839s) [upload=491693]: Got next segment. Inline: false
Dec 29 00:18:59.653008526 (560h49m59.524091562s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:18:59.653012754 (560h49m59.524095789s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:18:59.896335495 (560h49m59.76741853s) [seg_pos={0 0}, upload=491693]: Begin upload segment...
Dec 29 00:18:59.896379199 (560h49m59.767462234s) [seg_pos={0 0}, upload=491693]: Done begin upload segment.
Dec 29 00:18:59.896403324 (560h49m59.767486359s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896446496 (560h49m59.767529531s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896889490 (560h49m59.767972525s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897319879 (560h49m59.768402924s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897790155 (560h49m59.76887319s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897976620 (560h49m59.769059655s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898432107 (560h49m59.769515142s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898810398 (560h49m59.769893433s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898855464 (560h49m59.769938509s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899149123 (560h49m59.770232168s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899442471 (560h49m59.770525516s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899481035 (560h49m59.77056407s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900098410 (560h49m59.771181445s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900704284 (560h49m59.77178733s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.902352012 (560h49m59.773435047s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.904893571 (560h49m59.775976606s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:19:01.612870242 (560h50m1.483953287s) [upload=491693]: Next returned nil segment
Dec 29 00:19:01.612883587 (560h50m1.483966632s) [upload=491693]: Waiting for error group managing segments...
Dec 29 00:19:01.898642312 (560h50m1.769725367s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.900185751 (560h50m1.771268796s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913716255 (560h50m1.78479931s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913874717 (560h50m1.784957762s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918213118 (560h50m1.789296163s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918585226 (560h50m1.789668271s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.922171755 (560h50m1.7932548s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.929091652 (560h50m1.800174697s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.955713725 (560h50m1.82679677s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.123035108 (560h50m1.994118163s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.171790195 (560h50m2.04287324s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.174981291 (560h50m2.046064336s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202338894 (560h50m2.073421939s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202365354 (560h50m2.073448399s) [seg_pos={0 0}, upload=491693]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:19:02.202646460 (560h50m2.073729505s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202666839 (560h50m2.073749884s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202817105 (560h50m2.07390014s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202823577 (560h50m2.073906612s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202832995 (560h50m2.07391604s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202846781 (560h50m2.073929816s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202869164 (560h50m2.073952199s) [seg_pos={0 0}, upload=491693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:19:02.202951811 (560h50m2.074034856s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:19:02.202957111 (560h50m2.074040156s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:19:02.500241293 (560h50m2.371324348s) [upload=491693]: Done uploading segments...
Dec 29 00:19:02.500267042 (560h50m2.371350087s) [upload=491693]: Upload finished. err: <nil>
Duration: 6.157627345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.........................................X...X........X.X...]
duration: 
36.717330529s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:19:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 36.71735249s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [....X......X...X.............................X...X.X...X..X.]
duration: 
6.117038054s
trace: 
https://jaeger.dev.storj.io/trace/a8ef1a14dfa7309
FAIL (2025-12-29 00:19:42 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ) context deadline exceeded
======LOGS======
Dec 29 00:18:59.652723023 (560h49m59.523806058s) [upload=491693]: Starting upload
Dec 29 00:18:59.652774981 (560h49m59.523858017s) [upload=491693]: Done starting upload
Dec 29 00:18:59.652944194 (560h49m59.524027229s) [upload=491693]: Uploading segments...
Dec 29 00:18:59.652980803 (560h49m59.524063839s) [upload=491693]: Got next segment. Inline: false
Dec 29 00:18:59.653008526 (560h49m59.524091562s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:18:59.653012754 (560h49m59.524095789s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:18:59.896335495 (560h49m59.76741853s) [seg_pos={0 0}, upload=491693]: Begin upload segment...
Dec 29 00:18:59.896379199 (560h49m59.767462234s) [seg_pos={0 0}, upload=491693]: Done begin upload segment.
Dec 29 00:18:59.896403324 (560h49m59.767486359s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896446496 (560h49m59.767529531s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896889490 (560h49m59.767972525s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897319879 (560h49m59.768402924s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897790155 (560h49m59.76887319s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897976620 (560h49m59.769059655s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898432107 (560h49m59.769515142s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898810398 (560h49m59.769893433s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898855464 (560h49m59.769938509s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899149123 (560h49m59.770232168s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899442471 (560h49m59.770525516s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899481035 (560h49m59.77056407s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900098410 (560h49m59.771181445s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900704284 (560h49m59.77178733s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.902352012 (560h49m59.773435047s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.904893571 (560h49m59.775976606s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:19:01.612870242 (560h50m1.483953287s) [upload=491693]: Next returned nil segment
Dec 29 00:19:01.612883587 (560h50m1.483966632s) [upload=491693]: Waiting for error group managing segments...
Dec 29 00:19:01.898642312 (560h50m1.769725367s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.900185751 (560h50m1.771268796s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913716255 (560h50m1.78479931s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913874717 (560h50m1.784957762s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918213118 (560h50m1.789296163s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918585226 (560h50m1.789668271s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.922171755 (560h50m1.7932548s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.929091652 (560h50m1.800174697s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.955713725 (560h50m1.82679677s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.123035108 (560h50m1.994118163s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.171790195 (560h50m2.04287324s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.174981291 (560h50m2.046064336s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202338894 (560h50m2.073421939s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202365354 (560h50m2.073448399s) [seg_pos={0 0}, upload=491693]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:19:02.202646460 (560h50m2.073729505s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202666839 (560h50m2.073749884s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202817105 (560h50m2.07390014s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202823577 (560h50m2.073906612s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202832995 (560h50m2.07391604s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202846781 (560h50m2.073929816s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202869164 (560h50m2.073952199s) [seg_pos={0 0}, upload=491693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:19:02.202951811 (560h50m2.074034856s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:19:02.202957111 (560h50m2.074040156s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:19:02.500241293 (560h50m2.371324348s) [upload=491693]: Done uploading segments...
Dec 29 00:19:02.500267042 (560h50m2.371350087s) [upload=491693]: Upload finished. err: <nil>
Duration: 6.11706265s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [.....X......X...X.............................X...X.X...X..X]
duration: 
30.003437645s
trace: 
https://jaeger.dev.storj.io/trace/a8ef1a14dfa7309
FAIL (2025-12-29 00:19:28 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG, Piece ID: CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ) context deadline exceeded
======LOGS======
Dec 29 00:18:59.652723023 (560h49m59.523806058s) [upload=491693]: Starting upload
Dec 29 00:18:59.652774981 (560h49m59.523858017s) [upload=491693]: Done starting upload
Dec 29 00:18:59.652944194 (560h49m59.524027229s) [upload=491693]: Uploading segments...
Dec 29 00:18:59.652980803 (560h49m59.524063839s) [upload=491693]: Got next segment. Inline: false
Dec 29 00:18:59.653008526 (560h49m59.524091562s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:18:59.653012754 (560h49m59.524095789s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:18:59.896335495 (560h49m59.76741853s) [seg_pos={0 0}, upload=491693]: Begin upload segment...
Dec 29 00:18:59.896379199 (560h49m59.767462234s) [seg_pos={0 0}, upload=491693]: Done begin upload segment.
Dec 29 00:18:59.896403324 (560h49m59.767486359s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896446496 (560h49m59.767529531s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.896889490 (560h49m59.767972525s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897319879 (560h49m59.768402924s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897790155 (560h49m59.76887319s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.897976620 (560h49m59.769059655s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898432107 (560h49m59.769515142s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898810398 (560h49m59.769893433s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.898855464 (560h49m59.769938509s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899149123 (560h49m59.770232168s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899442471 (560h49m59.770525516s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.899481035 (560h49m59.77056407s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900098410 (560h49m59.771181445s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.900704284 (560h49m59.77178733s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.902352012 (560h49m59.773435047s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:18:59.904893571 (560h49m59.775976606s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Uploading piece...
Dec 29 00:19:01.612870242 (560h50m1.483953287s) [upload=491693]: Next returned nil segment
Dec 29 00:19:01.612883587 (560h50m1.483966632s) [upload=491693]: Waiting for error group managing segments...
Dec 29 00:19:01.898642312 (560h50m1.769725367s) [noise=true , address=145.79.227.19:4077   , piece_id=O4NPPM5C65W6FP6TMRBC75SJVWZJ67Z4O6FYCB65BTKGA7S63POA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.900185751 (560h50m1.771268796s) [noise=true , address=145.79.227.16:4909   , piece_id=WQT2SCXU6PBYVNPLFCFGMK6KBF2AE3OZB6YN52GXHXPPO4LA3VJA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913716255 (560h50m1.78479931s) [noise=true , address=145.79.227.37:4567   , piece_id=HZNVTAQDFAE62PYWQV4UOMTHE7O6X5JZFHE5IF7D5UNQQXAYXO6A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.913874717 (560h50m1.784957762s) [noise=true , address=145.79.227.41:4085   , piece_id=IQ2JEVJD7HIZDL3ZXJHIX6MJS6EJA6E3L7RO534LHOFBCNDDMHCQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918213118 (560h50m1.789296163s) [noise=true , address=145.79.227.46:4603   , piece_id=DCWYDN6FOEDPRADRHX6G5I46BHYAUGX7A2A5UEZRTI2BKM4N7X7A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.918585226 (560h50m1.789668271s) [noise=true , address=145.79.227.15:4874   , piece_id=V4DM7PMIF7VPE6SHIA3OFAHF6NRGEWKGL625Y3QBVX5KBQZX3RQA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.922171755 (560h50m1.7932548s) [noise=true , address=145.79.227.11:4730   , piece_id=THE25EEP34YYKUP2IM6TONC6JALK4YB2X3TAOURW6O65MMLYOTSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.929091652 (560h50m1.800174697s) [noise=true , address=145.79.227.48:4301   , piece_id=46LKI3G3S7JDCZB3JZFVAL5OHZX3ESISFTBKP74DDXHV2UA2LMAQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:01.955713725 (560h50m1.82679677s) [noise=true , address=145.79.227.36:4519   , piece_id=OYEPCCAAO2M2BB7XXRLYVZSEINOCMBIJCLYENT5XZTE3HWBWOAGA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.123035108 (560h50m1.994118163s) [noise=true , address=77.90.31.17:4770     , piece_id=CMQ6PFWA23OSHZY4JPRNWEVTOVQ6VKVLHJZGDRUU6VDLZ2MFT4FQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.171790195 (560h50m2.04287324s) [noise=true , address=45.91.116.5:4298     , piece_id=ROVNCBNZMKWNGXWTPP4L53QS3SWVVO2VRVSCEPVBIIQAT32JDANA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.174981291 (560h50m2.046064336s) [noise=true , address=45.91.116.10:4081    , piece_id=CCF55WH352JWC7BKE643JMINSAAYAJ2MJ3SMPPM5F3NZXDFR5EEA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202338894 (560h50m2.073421939s) [noise=true , address=45.91.116.8:4045     , piece_id=JMHY6NKI6XPA3OAFJJRTZRUGDTICCKUJEYPMEBVY37A5N6V5XUSQ, seg_pos={0 0}, upload=491693]: Done uploading piece. err: <nil>
Dec 29 00:19:02.202365354 (560h50m2.073448399s) [seg_pos={0 0}, upload=491693]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:19:02.202646460 (560h50m2.073729505s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202666839 (560h50m2.073749884s) [noise=true , address=45.91.116.9:4261     , piece_id=AU6Z4CDBOJDEQM4G3RTDBRRIAZOUWY4WE3G3TTT5W7B6CKBQN3LA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202817105 (560h50m2.07390014s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202823577 (560h50m2.073906612s) [noise=true , address=77.90.31.11:4555     , piece_id=TNKIKE36YTGYLJHIDDYEFHXYFPHOAREGOICQPMHZGX4IO4GZA7ZA, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202832995 (560h50m2.07391604s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:19:02.202846781 (560h50m2.073929816s) [noise=true , address=145.79.227.44:4193   , piece_id=HNFDOHVLGMGVTBQM2VMG5KETJUS7K7QBR7QSC5BURXNDJOMKVB3A, seg_pos={0 0}, upload=491693]: Long tail cancellation due to optimal threshold
Dec 29 00:19:02.202869164 (560h50m2.073952199s) [seg_pos={0 0}, upload=491693]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:19:02.202951811 (560h50m2.074034856s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:19:02.202957111 (560h50m2.074040156s) [upload=491693]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:19:02.500241293 (560h50m2.371324348s) [upload=491693]: Done uploading segments...
Dec 29 00:19:02.500267042 (560h50m2.371350087s) [upload=491693]: Upload finished. err: <nil>
Duration: 30.003457804s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..........................................X...X........X.X..]
duration: 
41.552651505s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:18:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 41.552668487s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [X.....X......X...X.............................X...X.X...X..]
duration: 
5.835196635s
trace: 
https://jaeger.dev.storj.io/trace/580656badddd6fac
FAIL (2025-12-29 00:18:46 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q) context deadline exceeded
======LOGS======
Dec 29 00:17:14.851806328 (560h48m14.722889363s) [upload=491668]: Starting upload
Dec 29 00:17:14.851917269 (560h48m14.723000324s) [upload=491668]: Done starting upload
Dec 29 00:17:14.852056624 (560h48m14.723139659s) [upload=491668]: Uploading segments...
Dec 29 00:17:14.852230876 (560h48m14.723313921s) [upload=491668]: Got next segment. Inline: false
Dec 29 00:17:14.852266725 (560h48m14.72334977s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:17:14.852272566 (560h48m14.723355601s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:17:15.098383516 (560h48m14.969466561s) [seg_pos={0 0}, upload=491668]: Begin upload segment...
Dec 29 00:17:15.098425285 (560h48m14.96950832s) [seg_pos={0 0}, upload=491668]: Done begin upload segment.
Dec 29 00:17:15.098462476 (560h48m14.969545521s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.098538512 (560h48m14.969621547s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099114859 (560h48m14.970197904s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099574454 (560h48m14.970657499s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099993511 (560h48m14.971076546s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100452647 (560h48m14.971535692s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100911230 (560h48m14.971994275s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102461822 (560h48m14.973544857s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102873185 (560h48m14.97395623s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.104037643 (560h48m14.975120688s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105231616 (560h48m14.976314651s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105567807 (560h48m14.976650842s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.106672590 (560h48m14.977755625s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.107539581 (560h48m14.978622616s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.108244703 (560h48m14.979327798s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.109426634 (560h48m14.980509679s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:16.811319582 (560h48m16.682402627s) [upload=491668]: Next returned nil segment
Dec 29 00:17:16.811356522 (560h48m16.682439557s) [upload=491668]: Waiting for error group managing segments...
Dec 29 00:17:17.098704564 (560h48m16.969787609s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.098752986 (560h48m16.969836021s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.099319244 (560h48m16.970402279s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109692176 (560h48m16.980775221s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109905772 (560h48m16.980988817s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.119468632 (560h48m16.990551667s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.125135762 (560h48m16.996218797s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.126883821 (560h48m16.997966866s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.128813375 (560h48m16.99989641s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.135386011 (560h48m17.006469056s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.289294875 (560h48m17.16037792s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.398905209 (560h48m17.269988254s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406217242 (560h48m17.277300277s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406239244 (560h48m17.277322279s) [seg_pos={0 0}, upload=491668]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:17:17.406437712 (560h48m17.277520747s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406444585 (560h48m17.27752762s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406560525 (560h48m17.27764356s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406581866 (560h48m17.277664901s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406664653 (560h48m17.277747698s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406671157 (560h48m17.277754192s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406694330 (560h48m17.277777365s) [seg_pos={0 0}, upload=491668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:17:17.406763542 (560h48m17.277846587s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:17:17.406768591 (560h48m17.277851626s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:17:17.711869276 (560h48m17.582952311s) [upload=491668]: Done uploading segments...
Dec 29 00:17:17.711888552 (560h48m17.582971587s) [upload=491668]: Upload finished. err: <nil>
Duration: 5.835224458s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...........................................X...X........X.X.]
duration: 
32.857563075s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:18:17 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 32.857586639s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [.X.....X......X...X.............................X...X.X...X.]
duration: 
6.126286278s
trace: 
https://jaeger.dev.storj.io/trace/580656badddd6fac
FAIL (2025-12-29 00:17:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q) context deadline exceeded
======LOGS======
Dec 29 00:17:14.851806328 (560h48m14.722889363s) [upload=491668]: Starting upload
Dec 29 00:17:14.851917269 (560h48m14.723000324s) [upload=491668]: Done starting upload
Dec 29 00:17:14.852056624 (560h48m14.723139659s) [upload=491668]: Uploading segments...
Dec 29 00:17:14.852230876 (560h48m14.723313921s) [upload=491668]: Got next segment. Inline: false
Dec 29 00:17:14.852266725 (560h48m14.72334977s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:17:14.852272566 (560h48m14.723355601s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:17:15.098383516 (560h48m14.969466561s) [seg_pos={0 0}, upload=491668]: Begin upload segment...
Dec 29 00:17:15.098425285 (560h48m14.96950832s) [seg_pos={0 0}, upload=491668]: Done begin upload segment.
Dec 29 00:17:15.098462476 (560h48m14.969545521s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.098538512 (560h48m14.969621547s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099114859 (560h48m14.970197904s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099574454 (560h48m14.970657499s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099993511 (560h48m14.971076546s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100452647 (560h48m14.971535692s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100911230 (560h48m14.971994275s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102461822 (560h48m14.973544857s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102873185 (560h48m14.97395623s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.104037643 (560h48m14.975120688s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105231616 (560h48m14.976314651s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105567807 (560h48m14.976650842s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.106672590 (560h48m14.977755625s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.107539581 (560h48m14.978622616s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.108244703 (560h48m14.979327798s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.109426634 (560h48m14.980509679s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:16.811319582 (560h48m16.682402627s) [upload=491668]: Next returned nil segment
Dec 29 00:17:16.811356522 (560h48m16.682439557s) [upload=491668]: Waiting for error group managing segments...
Dec 29 00:17:17.098704564 (560h48m16.969787609s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.098752986 (560h48m16.969836021s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.099319244 (560h48m16.970402279s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109692176 (560h48m16.980775221s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109905772 (560h48m16.980988817s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.119468632 (560h48m16.990551667s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.125135762 (560h48m16.996218797s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.126883821 (560h48m16.997966866s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.128813375 (560h48m16.99989641s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.135386011 (560h48m17.006469056s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.289294875 (560h48m17.16037792s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.398905209 (560h48m17.269988254s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406217242 (560h48m17.277300277s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406239244 (560h48m17.277322279s) [seg_pos={0 0}, upload=491668]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:17:17.406437712 (560h48m17.277520747s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406444585 (560h48m17.27752762s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406560525 (560h48m17.27764356s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406581866 (560h48m17.277664901s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406664653 (560h48m17.277747698s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406671157 (560h48m17.277754192s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406694330 (560h48m17.277777365s) [seg_pos={0 0}, upload=491668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:17:17.406763542 (560h48m17.277846587s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:17:17.406768591 (560h48m17.277851626s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:17:17.711869276 (560h48m17.582952311s) [upload=491668]: Done uploading segments...
Dec 29 00:17:17.711888552 (560h48m17.582971587s) [upload=491668]: Upload finished. err: <nil>
Duration: 6.12631814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [..X.....X......X...X.............................X...X.X...X]
duration: 
30.003180168s
trace: 
https://jaeger.dev.storj.io/trace/580656badddd6fac
FAIL (2025-12-29 00:17:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q) context deadline exceeded
======LOGS======
Dec 29 00:17:14.851806328 (560h48m14.722889363s) [upload=491668]: Starting upload
Dec 29 00:17:14.851917269 (560h48m14.723000324s) [upload=491668]: Done starting upload
Dec 29 00:17:14.852056624 (560h48m14.723139659s) [upload=491668]: Uploading segments...
Dec 29 00:17:14.852230876 (560h48m14.723313921s) [upload=491668]: Got next segment. Inline: false
Dec 29 00:17:14.852266725 (560h48m14.72334977s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:17:14.852272566 (560h48m14.723355601s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:17:15.098383516 (560h48m14.969466561s) [seg_pos={0 0}, upload=491668]: Begin upload segment...
Dec 29 00:17:15.098425285 (560h48m14.96950832s) [seg_pos={0 0}, upload=491668]: Done begin upload segment.
Dec 29 00:17:15.098462476 (560h48m14.969545521s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.098538512 (560h48m14.969621547s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099114859 (560h48m14.970197904s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099574454 (560h48m14.970657499s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.099993511 (560h48m14.971076546s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100452647 (560h48m14.971535692s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.100911230 (560h48m14.971994275s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102461822 (560h48m14.973544857s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.102873185 (560h48m14.97395623s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.104037643 (560h48m14.975120688s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105231616 (560h48m14.976314651s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.105567807 (560h48m14.976650842s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.106672590 (560h48m14.977755625s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.107539581 (560h48m14.978622616s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.108244703 (560h48m14.979327798s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:15.109426634 (560h48m14.980509679s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Uploading piece...
Dec 29 00:17:16.811319582 (560h48m16.682402627s) [upload=491668]: Next returned nil segment
Dec 29 00:17:16.811356522 (560h48m16.682439557s) [upload=491668]: Waiting for error group managing segments...
Dec 29 00:17:17.098704564 (560h48m16.969787609s) [noise=true , address=145.79.227.11:4729   , piece_id=GWZ4576GGTQZ7FAVAEGEV6THPFY7PZXK4K5G2YBHTPCF6D3JW7QA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.098752986 (560h48m16.969836021s) [noise=true , address=145.79.227.15:4875   , piece_id=II4QYFCMO7AHOPGMAITL2TELKHMYBHPKYMYSC5PUPELVXUA4WHRQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.099319244 (560h48m16.970402279s) [noise=true , address=145.79.227.36:4517   , piece_id=7S2EA2VOEXRFPAMEGP3YVN663LMXONOUPPJPAJVSMZJLYQNFUC6A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109692176 (560h48m16.980775221s) [noise=true , address=145.79.227.19:4077   , piece_id=YGMDYQ6CSNA2VVD7MRYNPWIVR44RXWIV2JKAAG4K2FRPO6XMRQMA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.109905772 (560h48m16.980988817s) [noise=true , address=145.79.227.47:4267   , piece_id=SAHFVHZHJUVZ7VFWFSLFD6HOFPFXY3VGURE5H65QHH3NIXIEWOZA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.119468632 (560h48m16.990551667s) [noise=true , address=145.79.227.26:4653   , piece_id=OPOIW2GRD2SBERKK4PCOGFPCZGCS3PYMECZXBUB3ZPZW55LJXNEQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.125135762 (560h48m16.996218797s) [noise=true , address=145.79.227.39:4014   , piece_id=7OAZGFKQN4RFFDZN7VLY3P4DEWXWUQLKVXXBZ2RZLPCWXR4GAZEA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.126883821 (560h48m16.997966866s) [noise=true , address=145.79.227.48:4302   , piece_id=2YRS2J5L3OTOTUJSDT3OEMSACXXL252IS5E74SVR42QS3DCXAY4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.128813375 (560h48m16.99989641s) [noise=true , address=145.79.227.50:4375   , piece_id=SZ3SMLTHGG33RXBOULDMNNDHKUDJGZNVFYEMWPYZCJST52IZ36QQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.135386011 (560h48m17.006469056s) [noise=true , address=145.79.227.21:4473   , piece_id=T7PDPH5TITM6KDIEZ4WUSMFOYYPXHCNIYDQM7RPFTEA72DP3V52A, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.289294875 (560h48m17.16037792s) [noise=true , address=77.90.31.19:4844     , piece_id=WGF5BQ55HYHAGZ7MVJ3ZCUAD2YEZ3UBZOHT4DDLZBXYATWZUND6Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.398905209 (560h48m17.269988254s) [noise=true , address=45.91.116.10:4083    , piece_id=4WBXMRYGNKH7A6YFIB3FXJ3ELJNAXTOJHUXXNOPYR6I3PODMDV4Q, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406217242 (560h48m17.277300277s) [noise=true , address=77.90.31.10:4518     , piece_id=AM547D6DMAHDKTP6J6I4V62XBQSC2G4ISKTYXW2R7PODFUM3WCUQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: <nil>
Dec 29 00:17:17.406239244 (560h48m17.277322279s) [seg_pos={0 0}, upload=491668]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:17:17.406437712 (560h48m17.277520747s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406444585 (560h48m17.27752762s) [noise=true , address=145.79.227.33:4869   , piece_id=QVHCW6GNKPLEF6J3PWVETYTZGVXEJDFNZWRPADNKXXRXZIRQCZAA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406560525 (560h48m17.27764356s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406581866 (560h48m17.277664901s) [noise=true , address=45.91.116.14:4190    , piece_id=N5HEK3BQGBNXBRBR3VSHCKNQ2ISVPVGJ2BOIYU2EJ3JZDVGM5MYQ, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406664653 (560h48m17.277747698s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 29 00:17:17.406671157 (560h48m17.277754192s) [noise=true , address=45.91.116.11:4118    , piece_id=55AZPGNUUPIHKAG7YQVNLO6TO7ITBGFHUPKVIWPXFQNZ5AF3UQKA, seg_pos={0 0}, upload=491668]: Long tail cancellation due to optimal threshold
Dec 29 00:17:17.406694330 (560h48m17.277777365s) [seg_pos={0 0}, upload=491668]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:17:17.406763542 (560h48m17.277846587s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:17:17.406768591 (560h48m17.277851626s) [upload=491668]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:17:17.711869276 (560h48m17.582952311s) [upload=491668]: Done uploading segments...
Dec 29 00:17:17.711888552 (560h48m17.582971587s) [upload=491668]: Upload finished. err: <nil>
Duration: 30.003203942s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [............................................X...X........X.X]
duration: 
1m0.004006998s
trace: 
https://jaeger.dev.storj.io/trace/34a6dd67ba0e5d39
FAIL (2025-12-29 00:17:40 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
714s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457310565 (560h47m41.32839361s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.457886532 (560h47m41.328969567s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458289749 (560h47m41.329372784s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.458747851 (560h47m41.329830886s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459158834 (560h47m41.330241879s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459549358 (560h47m41.330632393s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.459955221 (560h47m41.331038256s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460338280 (560h47m41.331421315s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.460744885 (560h47m41.33182792s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461723167 (560h47m41.332806212s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.461961321 (560h47m41.333044356s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.463221530 (560h47m41.334304566s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.464652265 (560h47m41.3357353s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.466458825 (560h47m41.33754187s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:41.469979879 (560h47m41.341062914s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Uploading piece...
Dec 29 00:16:42.940523414 (560h47m42.811606449s) [upload=491661]: Next returned nil segment
Dec 29 00:16:42.940533884 (560h47m42.811616919s) [upload=491661]: Waiting for error group managing segments...
Dec 29 00:16:43.348379100 (560h47m43.219462135s) [noise=true , address=145.79.227.14:4872   , piece_id=MYII2TGLSOZEUAWZLVVAEIV6KFWPO6XWHHKZNQ5ZSDGIPBOB5OMA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.353900272 (560h47m43.224983307s) [noise=true , address=145.79.227.24:4612   , piece_id=6ZTZX7QPQFV54ADIFWU5GPRN55SHEVBY5PG7GLRU6VWT7G6KJU3A, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.366484214 (560h47m43.237567259s) [noise=true , address=145.79.227.25:4646   , piece_id=ACLYCOQHA2U2PXRWDE37CFJQE75QSYHV53JTD774KK6GMQ2XQLBQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.503787639 (560h47m43.374870674s) [noise=true , address=77.90.31.13:4648     , piece_id=P5QLMG4JWPIDXWX3KX7JYT4VYFJAPFYW6U3TQ3ISPALD5Q4LQ5CA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.555237688 (560h47m43.426320733s) [noise=true , address=45.91.116.11:4146    , piece_id=TE652BYNSWCQ5Y3JGAM6OJA7KIXIMSKYEPPE47IBIOXXTEX5K6SQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.590911715 (560h47m43.46199475s) [noise=true , address=77.90.31.16:4752     , piece_id=JLFS7ADNZZEYIJYRIMJ5HGFFOIRXTGCQQQ3WMERL3KY6YJXBVTNA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.598615686 (560h47m43.469698721s) [noise=true , address=77.90.31.11:4558     , piece_id=6J7C5P3THIRUSO6XMXFZP7RPJY5ALQKGLVLFOZIPXTB655TK2IOA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.694599091 (560h47m43.565682126s) [noise=true , address=145.79.227.19:4088   , piece_id=MBHVWPJX53JU7UZMA4JQYLP6NIBYFWQXP7UUWOKSNLK4Q4YWCUKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:43.715008664 (560h47m43.58609171s) [noise=true , address=45.91.116.12:4234    , piece_id=SGLVOSFJKFIUZ7IFNQCN6KLKVH5UHQEPKWQNZBYVKE5ZZ25BJMGA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.259833879 (560h47m44.130916924s) [noise=true , address=145.79.227.30:4786   , piece_id=LA2TD6LJC6RJIUDMHXOK3LEMGPPBXIK5COV4G2YMX7ENL5QYIRPA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.270639214 (560h47m44.141722249s) [noise=true , address=145.79.227.43:4167   , piece_id=XDEZLYFPTIPU2WK3CB3JOXBQWQOGVBY5V4FB65E3HJ4XA2K2FMSA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.271845310 (560h47m44.142928345s) [noise=true , address=145.79.227.30:4796   , piece_id=MFWKMEVMNR26EL2WJ5LNGDR6JUO5L4E5UXJEEORHFT74WTSCGK2Q, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.297505942 (560h47m44.168588987s) [noise=true , address=145.79.227.44:4224   , piece_id=R6D5MUMGUICNNMXVKCC6FPA6HOYWNGWFBM3KIZFVMIK2OYJRA6UA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328744514 (560h47m44.199827559s) [noise=true , address=145.79.227.49:4357   , piece_id=Y5RXSTX66Z4L5LQJNB3ACHHX3KA4NP7IOOSHQN7RV2W7JN3SGOJQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: <nil>
Dec 29 00:16:44.328760774 (560h47m44.199843809s) [seg_pos={0 0}, upload=491661]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:16:44.328915139 (560h47m44.199998174s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.328921260 (560h47m44.200004305s) [noise=true , address=77.90.31.18:4824     , piece_id=VS3IKLEVU5MMPU5BSDL2KMWAW2TTHRDLYAS3MI3DMDT2SIMJYPKQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.328932161 (560h47m44.200015196s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58000->64.49.13.169:4707: use of closed network connection
Dec 29 00:16:44.328940086 (560h47m44.200023121s) [noise=true , address=64.49.13.169:4707    , piece_id=H2OXVZ6ELAOXTQ7IY3JYUTTPVHWQYIDD2NUM3IP3HD7IGGXUUNEA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329029266 (560h47m44.200112311s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1BPYCUuXmyUAZWEgWk2GWTnmT4Sh2pvnvHLHaFaArX3iSvkrnq): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329034175 (560h47m44.20011721s) [noise=true , address=145.79.227.40:4067   , piece_id=5ZK6Z3737BPT6WUP6NHPTFOULLSDWKILNIOWEDQLLVQ7U2M4YWDA, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329132563 (560h47m44.200215598s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU): protocol: expected piece hash; context canceled; EOF
Dec 29 00:16:44.329137792 (560h47m44.200220828s) [noise=true , address=145.79.227.33:4897   , piece_id=HQDAY4QVX7UQ6FT33DHDM3ENEQYH6DCU4DURXAWEQIS6ETQZ73BQ, seg_pos={0 0}, upload=491661]: Long tail cancellation due to optimal threshold
Dec 29 00:16:44.329185723 (560h47m44.200268758s) [seg_pos={0 0}, upload=491661]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:16:44.329248914 (560h47m44.200331949s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:16:44.329254695 (560h47m44.20033773s) [upload=491661]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:16:44.629607259 (560h47m44.500690294s) [upload=491661]: Done uploading segments...
Dec 29 00:16:44.629627588 (560h47m44.500710623s) [upload=491661]: Upload finished. err: <nil>
Duration: 1m0.00402996s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [...X.....X......X...X.............................X...X.X...]
duration: 
13.894519885s
trace: 
https://jaeger.dev.storj.io/trace/27d4807dc8c85859
FAIL (2025-12-29 00:16:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA) context deadline exceeded
======LOGS======
Dec 29 00:13:56.535571445 (560h44m56.40665448s) [upload=491623]: Starting upload
Dec 29 00:13:56.535621891 (560h44m56.406704926s) [upload=491623]: Done starting upload
Dec 29 00:13:56.535766195 (560h44m56.40684923s) [upload=491623]: Uploading segments...
Dec 29 00:13:56.535884261 (560h44m56.406967296s) [upload=491623]: Got next segment. Inline: false
Dec 29 00:13:56.535950155 (560h44m56.407033191s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:13:56.535956669 (560h44m56.407039704s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:13:56.784976337 (560h44m56.656059382s) [seg_pos={0 0}, upload=491623]: Begin upload segment...
Dec 29 00:13:56.785028246 (560h44m56.656111291s) [seg_pos={0 0}, upload=491623]: Done begin upload segment.
Dec 29 00:13:56.785058503 (560h44m56.656141538s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785067951 (560h44m56.656150996s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785837517 (560h44m56.656920552s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785879817 (560h44m56.656962862s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786207791 (560h44m56.657290826s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786482865 (560h44m56.6575659s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786771245 (560h44m56.65785428s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787031961 (560h44m56.658114996s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787670256 (560h44m56.658753291s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788099514 (560h44m56.659182549s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788476843 (560h44m56.659559878s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788785009 (560h44m56.659868054s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789180001 (560h44m56.660263036s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789713827 (560h44m56.660796863s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.790063784 (560h44m56.661146819s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.791131708 (560h44m56.662214743s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:58.502036914 (560h44m58.373119949s) [upload=491623]: Next returned nil segment
Dec 29 00:13:58.502054107 (560h44m58.373137152s) [upload=491623]: Waiting for error group managing segments...
Dec 29 00:13:58.789868406 (560h44m58.660951441s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.795195310 (560h44m58.666278355s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.796611455 (560h44m58.667694491s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.800487035 (560h44m58.67157007s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.807920891 (560h44m58.679003936s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.808728328 (560h44m58.679811363s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.818637967 (560h44m58.689721002s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.992291990 (560h44m58.863375025s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.028644179 (560h44m58.899727214s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.070650735 (560h44m58.94173377s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.089366112 (560h44m58.960449158s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.094008442 (560h44m58.965091477s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189151949 (560h44m59.060234984s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189169192 (560h44m59.060252227s) [seg_pos={0 0}, upload=491623]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:13:59.189361147 (560h44m59.060444182s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:13:59.189367199 (560h44m59.060450234s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189679433 (560h44m59.060762468s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:53002->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189694703 (560h44m59.060777738s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189710252 (560h44m59.060793287s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:49730->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189717145 (560h44m59.06080018s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189730892 (560h44m59.060813927s) [seg_pos={0 0}, upload=491623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:13:59.189791677 (560h44m59.060874712s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:13:59.189818598 (560h44m59.060901633s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:13:59.479474574 (560h44m59.350557609s) [upload=491623]: Done uploading segments...
Dec 29 00:13:59.479497958 (560h44m59.350581003s) [upload=491623]: Upload finished. err: <nil>
Duration: 13.894549341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.............................................X...X........X.]
duration: 
26.676268377s
trace: 
https://jaeger.dev.storj.io/trace/58f9ae34de0ef7cb
FAIL (2025-12-29 00:15:58 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
 piece_id=57GYLUWE6OVJXI36E6RVRMFM4I5PQEABMFR67APDAZJ7NKBYWZOQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.899096213 (560h45m32.770179258s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.900198482 (560h45m32.771281527s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.900393573 (560h45m32.771476608s) [noise=true , address=77.90.31.17:4778     , piece_id=WCXZWPWAWJ6CUTM7M4ZKKOWAPCBZ6D5RGDVAVYSBVLC3GJFNBW7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.902049176 (560h45m32.773132221s) [noise=true , address=77.90.31.10:4536     , piece_id=M7UEKGJDRZEMKUTJGUD55JF2JQ3ASB2VPJAPEF2T32OG3XRATLLQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.903603124 (560h45m32.774686159s) [noise=true , address=64.49.13.139:4473    , piece_id=LCFX3US2SYEWZYXT4GE7AMCH27274IGSGHLHEAZDDN6REL7PEC3Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.904602377 (560h45m32.775685422s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.905228038 (560h45m32.776311073s) [noise=true , address=145.79.227.29:4959   , piece_id=5N5Y7FAPV4LU2Q7Q4NIDMOXJNTYTGIYDGSE37BX6IBFP5H25ULWQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.906668050 (560h45m32.777751085s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.906982489 (560h45m32.778065524s) [noise=true , address=145.79.227.2:4141    , piece_id=GIK23WBLLYMJMKW2P7SFSN46S3YAOCIRDOTBQ6TBIUGG74MO6I7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.907759108 (560h45m32.778842143s) [noise=true , address=77.90.31.11:4584     , piece_id=6HDBM7AVDYYAKSJRSJ7ULKLA6SBTHZMNZRRQ56JI6L4KIQIF543Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.908734446 (560h45m32.779817481s) [noise=true , address=64.49.13.138:4492    , piece_id=ZIID7QJHFNVGHKHA34UR64PSX45QWOB3WQQBWRX6SVXFYIUDXE7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.909429409 (560h45m32.780512454s) [noise=true , address=145.79.227.52:4459   , piece_id=RRTKVNYIEW7OMRTUDTXONISYASE2LKRKJSCDOWHOZ7ZKUNWAVGYA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.910821389 (560h45m32.781904435s) [noise=true , address=77.90.31.12:4623     , piece_id=IT4KW4TQOBK5BC6ZZD4FYJ7QZ4V77QHORU7MPAQIFU32T6WNZ4EA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.913299017 (560h45m32.784382052s) [noise=true , address=145.79.227.44:4228   , piece_id=GF4GTTAJBMTZIAT56D3SLVTWRLDWXIYKPUHBEW2BDFG4UQM7WHMA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:34.231871126 (560h45m34.102954161s) [upload=491634]: Next returned nil segment
Dec 29 00:14:34.231888548 (560h45m34.102971584s) [upload=491634]: Waiting for error group managing segments...
Dec 29 00:14:34.579429457 (560h45m34.450512492s) [noise=true , address=64.49.13.139:4473    , piece_id=LCFX3US2SYEWZYXT4GE7AMCH27274IGSGHLHEAZDDN6REL7PEC3Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.644135616 (560h45m34.515218661s) [noise=true , address=64.49.13.138:4492    , piece_id=ZIID7QJHFNVGHKHA34UR64PSX45QWOB3WQQBWRX6SVXFYIUDXE7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.795608372 (560h45m34.666691417s) [noise=true , address=145.79.227.2:4141    , piece_id=GIK23WBLLYMJMKW2P7SFSN46S3YAOCIRDOTBQ6TBIUGG74MO6I7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.824849541 (560h45m34.695932576s) [noise=true , address=145.79.227.52:4459   , piece_id=RRTKVNYIEW7OMRTUDTXONISYASE2LKRKJSCDOWHOZ7ZKUNWAVGYA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.967797875 (560h45m34.83888091s) [noise=true , address=77.90.31.10:4536     , piece_id=M7UEKGJDRZEMKUTJGUD55JF2JQ3ASB2VPJAPEF2T32OG3XRATLLQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.035239674 (560h45m34.906322709s) [noise=true , address=77.90.31.17:4778     , piece_id=WCXZWPWAWJ6CUTM7M4ZKKOWAPCBZ6D5RGDVAVYSBVLC3GJFNBW7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.053262852 (560h45m34.924345887s) [noise=true , address=77.90.31.11:4584     , piece_id=6HDBM7AVDYYAKSJRSJ7ULKLA6SBTHZMNZRRQ56JI6L4KIQIF543Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.078025263 (560h45m34.949108298s) [noise=true , address=45.91.116.10:4103    , piece_id=BT2H5E3BNZTVFG5KCW6JKY3M24GLMREFO3RVMKT44PF2FCW6ZMMQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.094314160 (560h45m34.965397195s) [noise=true , address=77.90.31.12:4623     , piece_id=IT4KW4TQOBK5BC6ZZD4FYJ7QZ4V77QHORU7MPAQIFU32T6WNZ4EA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.654641042 (560h45m35.525724077s) [noise=true , address=145.79.227.43:4180   , piece_id=57GYLUWE6OVJXI36E6RVRMFM4I5PQEABMFR67APDAZJ7NKBYWZOQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.671621124 (560h45m35.542704159s) [noise=true , address=145.79.227.30:4780   , piece_id=GMORRA7XOBWAUSZFV5WK3T37MRHCVGSGFHDA2QURLXPO5MJ3S2ZA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.742331778 (560h45m35.613414813s) [noise=true , address=145.79.227.44:4228   , piece_id=GF4GTTAJBMTZIAT56D3SLVTWRLDWXIYKPUHBEW2BDFG4UQM7WHMA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.808952152 (560h45m35.680035187s) [noise=true , address=145.79.227.35:4498   , piece_id=RDGDYUFSTLOC3GI5QOAYQVXQTOALHXPJEKYQHUJTY55QTKTTTZCA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.809597071 (560h45m35.680680116s) [noise=true , address=145.79.227.29:4959   , piece_id=5N5Y7FAPV4LU2Q7Q4NIDMOXJNTYTGIYDGSE37BX6IBFP5H25ULWQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.809606999 (560h45m35.680690034s) [seg_pos={0 0}, upload=491634]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:14:35.809771853 (560h45m35.680854888s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 29 00:14:35.809778616 (560h45m35.680861651s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.809975671 (560h45m35.681058706s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: expected piece hash; context canceled; EOF
Dec 29 00:14:35.809982284 (560h45m35.681065319s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810088777 (560h45m35.681171822s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection
Dec 29 00:14:35.810094267 (560h45m35.681177302s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810205809 (560h45m35.681288844s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC): protocol: write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection; write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection
Dec 29 00:14:35.810214796 (560h45m35.681297831s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810234875 (560h45m35.68131791s) [seg_pos={0 0}, upload=491634]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:14:35.810309497 (560h45m35.681392532s) [upload=491634]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:14:35.810315448 (560h45m35.681398493s) [upload=491634]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:14:36.109047296 (560h45m35.980130341s) [upload=491634]: Done uploading segments...
Dec 29 00:14:36.109070410 (560h45m35.980153445s) [upload=491634]: Upload finished. err: <nil>
Duration: 26.676282543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [....X.....X......X...X.............................X...X.X..]
duration: 
5.76842063s
trace: 
https://jaeger.dev.storj.io/trace/27d4807dc8c85859
FAIL (2025-12-29 00:15:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA) context deadline exceeded
======LOGS======
Dec 29 00:13:56.535571445 (560h44m56.40665448s) [upload=491623]: Starting upload
Dec 29 00:13:56.535621891 (560h44m56.406704926s) [upload=491623]: Done starting upload
Dec 29 00:13:56.535766195 (560h44m56.40684923s) [upload=491623]: Uploading segments...
Dec 29 00:13:56.535884261 (560h44m56.406967296s) [upload=491623]: Got next segment. Inline: false
Dec 29 00:13:56.535950155 (560h44m56.407033191s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:13:56.535956669 (560h44m56.407039704s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:13:56.784976337 (560h44m56.656059382s) [seg_pos={0 0}, upload=491623]: Begin upload segment...
Dec 29 00:13:56.785028246 (560h44m56.656111291s) [seg_pos={0 0}, upload=491623]: Done begin upload segment.
Dec 29 00:13:56.785058503 (560h44m56.656141538s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785067951 (560h44m56.656150996s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785837517 (560h44m56.656920552s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785879817 (560h44m56.656962862s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786207791 (560h44m56.657290826s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786482865 (560h44m56.6575659s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786771245 (560h44m56.65785428s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787031961 (560h44m56.658114996s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787670256 (560h44m56.658753291s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788099514 (560h44m56.659182549s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788476843 (560h44m56.659559878s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788785009 (560h44m56.659868054s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789180001 (560h44m56.660263036s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789713827 (560h44m56.660796863s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.790063784 (560h44m56.661146819s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.791131708 (560h44m56.662214743s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:58.502036914 (560h44m58.373119949s) [upload=491623]: Next returned nil segment
Dec 29 00:13:58.502054107 (560h44m58.373137152s) [upload=491623]: Waiting for error group managing segments...
Dec 29 00:13:58.789868406 (560h44m58.660951441s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.795195310 (560h44m58.666278355s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.796611455 (560h44m58.667694491s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.800487035 (560h44m58.67157007s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.807920891 (560h44m58.679003936s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.808728328 (560h44m58.679811363s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.818637967 (560h44m58.689721002s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.992291990 (560h44m58.863375025s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.028644179 (560h44m58.899727214s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.070650735 (560h44m58.94173377s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.089366112 (560h44m58.960449158s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.094008442 (560h44m58.965091477s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189151949 (560h44m59.060234984s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189169192 (560h44m59.060252227s) [seg_pos={0 0}, upload=491623]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:13:59.189361147 (560h44m59.060444182s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:13:59.189367199 (560h44m59.060450234s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189679433 (560h44m59.060762468s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:53002->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189694703 (560h44m59.060777738s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189710252 (560h44m59.060793287s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:49730->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189717145 (560h44m59.06080018s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189730892 (560h44m59.060813927s) [seg_pos={0 0}, upload=491623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:13:59.189791677 (560h44m59.060874712s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:13:59.189818598 (560h44m59.060901633s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:13:59.479474574 (560h44m59.350557609s) [upload=491623]: Done uploading segments...
Dec 29 00:13:59.479497958 (560h44m59.350581003s) [upload=491623]: Upload finished. err: <nil>
Duration: 5.768440948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..............................................X...X........X]
duration: 
1m0.002716663s
trace: 
https://jaeger.dev.storj.io/trace/58f9ae34de0ef7cb
FAIL (2025-12-29 00:15:31 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
 piece_id=57GYLUWE6OVJXI36E6RVRMFM4I5PQEABMFR67APDAZJ7NKBYWZOQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.899096213 (560h45m32.770179258s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.900198482 (560h45m32.771281527s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.900393573 (560h45m32.771476608s) [noise=true , address=77.90.31.17:4778     , piece_id=WCXZWPWAWJ6CUTM7M4ZKKOWAPCBZ6D5RGDVAVYSBVLC3GJFNBW7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.902049176 (560h45m32.773132221s) [noise=true , address=77.90.31.10:4536     , piece_id=M7UEKGJDRZEMKUTJGUD55JF2JQ3ASB2VPJAPEF2T32OG3XRATLLQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.903603124 (560h45m32.774686159s) [noise=true , address=64.49.13.139:4473    , piece_id=LCFX3US2SYEWZYXT4GE7AMCH27274IGSGHLHEAZDDN6REL7PEC3Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.904602377 (560h45m32.775685422s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.905228038 (560h45m32.776311073s) [noise=true , address=145.79.227.29:4959   , piece_id=5N5Y7FAPV4LU2Q7Q4NIDMOXJNTYTGIYDGSE37BX6IBFP5H25ULWQ, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.906668050 (560h45m32.777751085s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.906982489 (560h45m32.778065524s) [noise=true , address=145.79.227.2:4141    , piece_id=GIK23WBLLYMJMKW2P7SFSN46S3YAOCIRDOTBQ6TBIUGG74MO6I7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.907759108 (560h45m32.778842143s) [noise=true , address=77.90.31.11:4584     , piece_id=6HDBM7AVDYYAKSJRSJ7ULKLA6SBTHZMNZRRQ56JI6L4KIQIF543Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.908734446 (560h45m32.779817481s) [noise=true , address=64.49.13.138:4492    , piece_id=ZIID7QJHFNVGHKHA34UR64PSX45QWOB3WQQBWRX6SVXFYIUDXE7Q, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.909429409 (560h45m32.780512454s) [noise=true , address=145.79.227.52:4459   , piece_id=RRTKVNYIEW7OMRTUDTXONISYASE2LKRKJSCDOWHOZ7ZKUNWAVGYA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.910821389 (560h45m32.781904435s) [noise=true , address=77.90.31.12:4623     , piece_id=IT4KW4TQOBK5BC6ZZD4FYJ7QZ4V77QHORU7MPAQIFU32T6WNZ4EA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:32.913299017 (560h45m32.784382052s) [noise=true , address=145.79.227.44:4228   , piece_id=GF4GTTAJBMTZIAT56D3SLVTWRLDWXIYKPUHBEW2BDFG4UQM7WHMA, seg_pos={0 0}, upload=491634]: Uploading piece...
Dec 29 00:14:34.231871126 (560h45m34.102954161s) [upload=491634]: Next returned nil segment
Dec 29 00:14:34.231888548 (560h45m34.102971584s) [upload=491634]: Waiting for error group managing segments...
Dec 29 00:14:34.579429457 (560h45m34.450512492s) [noise=true , address=64.49.13.139:4473    , piece_id=LCFX3US2SYEWZYXT4GE7AMCH27274IGSGHLHEAZDDN6REL7PEC3Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.644135616 (560h45m34.515218661s) [noise=true , address=64.49.13.138:4492    , piece_id=ZIID7QJHFNVGHKHA34UR64PSX45QWOB3WQQBWRX6SVXFYIUDXE7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.795608372 (560h45m34.666691417s) [noise=true , address=145.79.227.2:4141    , piece_id=GIK23WBLLYMJMKW2P7SFSN46S3YAOCIRDOTBQ6TBIUGG74MO6I7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.824849541 (560h45m34.695932576s) [noise=true , address=145.79.227.52:4459   , piece_id=RRTKVNYIEW7OMRTUDTXONISYASE2LKRKJSCDOWHOZ7ZKUNWAVGYA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:34.967797875 (560h45m34.83888091s) [noise=true , address=77.90.31.10:4536     , piece_id=M7UEKGJDRZEMKUTJGUD55JF2JQ3ASB2VPJAPEF2T32OG3XRATLLQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.035239674 (560h45m34.906322709s) [noise=true , address=77.90.31.17:4778     , piece_id=WCXZWPWAWJ6CUTM7M4ZKKOWAPCBZ6D5RGDVAVYSBVLC3GJFNBW7Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.053262852 (560h45m34.924345887s) [noise=true , address=77.90.31.11:4584     , piece_id=6HDBM7AVDYYAKSJRSJ7ULKLA6SBTHZMNZRRQ56JI6L4KIQIF543Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.078025263 (560h45m34.949108298s) [noise=true , address=45.91.116.10:4103    , piece_id=BT2H5E3BNZTVFG5KCW6JKY3M24GLMREFO3RVMKT44PF2FCW6ZMMQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.094314160 (560h45m34.965397195s) [noise=true , address=77.90.31.12:4623     , piece_id=IT4KW4TQOBK5BC6ZZD4FYJ7QZ4V77QHORU7MPAQIFU32T6WNZ4EA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.654641042 (560h45m35.525724077s) [noise=true , address=145.79.227.43:4180   , piece_id=57GYLUWE6OVJXI36E6RVRMFM4I5PQEABMFR67APDAZJ7NKBYWZOQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.671621124 (560h45m35.542704159s) [noise=true , address=145.79.227.30:4780   , piece_id=GMORRA7XOBWAUSZFV5WK3T37MRHCVGSGFHDA2QURLXPO5MJ3S2ZA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.742331778 (560h45m35.613414813s) [noise=true , address=145.79.227.44:4228   , piece_id=GF4GTTAJBMTZIAT56D3SLVTWRLDWXIYKPUHBEW2BDFG4UQM7WHMA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.808952152 (560h45m35.680035187s) [noise=true , address=145.79.227.35:4498   , piece_id=RDGDYUFSTLOC3GI5QOAYQVXQTOALHXPJEKYQHUJTY55QTKTTTZCA, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.809597071 (560h45m35.680680116s) [noise=true , address=145.79.227.29:4959   , piece_id=5N5Y7FAPV4LU2Q7Q4NIDMOXJNTYTGIYDGSE37BX6IBFP5H25ULWQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: <nil>
Dec 29 00:14:35.809606999 (560h45m35.680690034s) [seg_pos={0 0}, upload=491634]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:14:35.809771853 (560h45m35.680854888s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qaYFSpezuLAc5zJfAYruzad6nTpYFDpog7bmKMVTA8MVm76NB): protocol: expected piece hash; context canceled; EOF
Dec 29 00:14:35.809778616 (560h45m35.680861651s) [noise=true , address=145.79.227.34:4930   , piece_id=75U2B2KB42JC3BXZB3RGOG2BLBAETD5GDUK2SE4A3XQE6TCT246Q, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.809975671 (560h45m35.681058706s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ErG54kzXSK5hXDTYZpL6X3dg26sqW2kdM6CQMLp3qMMHteir1): protocol: expected piece hash; context canceled; EOF
Dec 29 00:14:35.809982284 (560h45m35.681065319s) [noise=true , address=145.79.227.49:4356   , piece_id=YUK62XWIRSLMB4QG466F5VOHYA4UHP6XKTTOZJKXFKZSRHNRQK5Q, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810088777 (560h45m35.681171822s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1hsbGK94RFwxic9F8NKp8kKvztegUhkNmYEor4QyLJmqF3seZf): protocol: write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection; write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57672->64.49.13.169:4710: use of closed network connection
Dec 29 00:14:35.810094267 (560h45m35.681177302s) [noise=true , address=64.49.13.169:4710    , piece_id=L2BBM5FEMBRBRDO2R22XHYGJCALA364CIPDRX3GIO6FOHMNWCLVQ, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810205809 (560h45m35.681288844s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Y16Z5H7jpsCyRzsAajF5AR3mURqTY8a4jgEr2xP4Yxy9W9kdC): protocol: write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection; write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36180->145.79.227.27:4720: use of closed network connection
Dec 29 00:14:35.810214796 (560h45m35.681297831s) [noise=true , address=145.79.227.27:4720   , piece_id=R342U7RJPLUHFG3G3NYFKJCSVIJIRNIIB2H2CGQ2EKSKMKODXG4A, seg_pos={0 0}, upload=491634]: Long tail cancellation due to optimal threshold
Dec 29 00:14:35.810234875 (560h45m35.68131791s) [seg_pos={0 0}, upload=491634]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:14:35.810309497 (560h45m35.681392532s) [upload=491634]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:14:35.810315448 (560h45m35.681398493s) [upload=491634]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:14:36.109047296 (560h45m35.980130341s) [upload=491634]: Done uploading segments...
Dec 29 00:14:36.109070410 (560h45m35.980153445s) [upload=491634]: Upload finished. err: <nil>
Duration: 1m0.00272553s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.....X.....X......X...X.............................X...X.X.]
duration: 
6.005491982s
trace: 
https://jaeger.dev.storj.io/trace/27d4807dc8c85859
FAIL (2025-12-29 00:14:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA) context deadline exceeded
======LOGS======
Dec 29 00:13:56.535571445 (560h44m56.40665448s) [upload=491623]: Starting upload
Dec 29 00:13:56.535621891 (560h44m56.406704926s) [upload=491623]: Done starting upload
Dec 29 00:13:56.535766195 (560h44m56.40684923s) [upload=491623]: Uploading segments...
Dec 29 00:13:56.535884261 (560h44m56.406967296s) [upload=491623]: Got next segment. Inline: false
Dec 29 00:13:56.535950155 (560h44m56.407033191s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:13:56.535956669 (560h44m56.407039704s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:13:56.784976337 (560h44m56.656059382s) [seg_pos={0 0}, upload=491623]: Begin upload segment...
Dec 29 00:13:56.785028246 (560h44m56.656111291s) [seg_pos={0 0}, upload=491623]: Done begin upload segment.
Dec 29 00:13:56.785058503 (560h44m56.656141538s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785067951 (560h44m56.656150996s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785837517 (560h44m56.656920552s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785879817 (560h44m56.656962862s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786207791 (560h44m56.657290826s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786482865 (560h44m56.6575659s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786771245 (560h44m56.65785428s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787031961 (560h44m56.658114996s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787670256 (560h44m56.658753291s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788099514 (560h44m56.659182549s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788476843 (560h44m56.659559878s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788785009 (560h44m56.659868054s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789180001 (560h44m56.660263036s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789713827 (560h44m56.660796863s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.790063784 (560h44m56.661146819s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.791131708 (560h44m56.662214743s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:58.502036914 (560h44m58.373119949s) [upload=491623]: Next returned nil segment
Dec 29 00:13:58.502054107 (560h44m58.373137152s) [upload=491623]: Waiting for error group managing segments...
Dec 29 00:13:58.789868406 (560h44m58.660951441s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.795195310 (560h44m58.666278355s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.796611455 (560h44m58.667694491s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.800487035 (560h44m58.67157007s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.807920891 (560h44m58.679003936s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.808728328 (560h44m58.679811363s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.818637967 (560h44m58.689721002s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.992291990 (560h44m58.863375025s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.028644179 (560h44m58.899727214s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.070650735 (560h44m58.94173377s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.089366112 (560h44m58.960449158s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.094008442 (560h44m58.965091477s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189151949 (560h44m59.060234984s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189169192 (560h44m59.060252227s) [seg_pos={0 0}, upload=491623]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:13:59.189361147 (560h44m59.060444182s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:13:59.189367199 (560h44m59.060450234s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189679433 (560h44m59.060762468s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:53002->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189694703 (560h44m59.060777738s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189710252 (560h44m59.060793287s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:49730->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189717145 (560h44m59.06080018s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189730892 (560h44m59.060813927s) [seg_pos={0 0}, upload=491623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:13:59.189791677 (560h44m59.060874712s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:13:59.189818598 (560h44m59.060901633s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:13:59.479474574 (560h44m59.350557609s) [upload=491623]: Done uploading segments...
Dec 29 00:13:59.479497958 (560h44m59.350581003s) [upload=491623]: Upload finished. err: <nil>
Duration: 6.005518361s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [X.....X.....X......X...X.............................X...X.X]
duration: 
30.002176875s
trace: 
https://jaeger.dev.storj.io/trace/27d4807dc8c85859
FAIL (2025-12-29 00:14:25 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12SnvkmRPw7yGAfbNNx6QeZMnpo6NuiVT2PyzBAHuhgMgsxWKWX, Piece ID: C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA) context deadline exceeded
======LOGS======
Dec 29 00:13:56.535571445 (560h44m56.40665448s) [upload=491623]: Starting upload
Dec 29 00:13:56.535621891 (560h44m56.406704926s) [upload=491623]: Done starting upload
Dec 29 00:13:56.535766195 (560h44m56.40684923s) [upload=491623]: Uploading segments...
Dec 29 00:13:56.535884261 (560h44m56.406967296s) [upload=491623]: Got next segment. Inline: false
Dec 29 00:13:56.535950155 (560h44m56.407033191s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:13:56.535956669 (560h44m56.407039704s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:13:56.784976337 (560h44m56.656059382s) [seg_pos={0 0}, upload=491623]: Begin upload segment...
Dec 29 00:13:56.785028246 (560h44m56.656111291s) [seg_pos={0 0}, upload=491623]: Done begin upload segment.
Dec 29 00:13:56.785058503 (560h44m56.656141538s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785067951 (560h44m56.656150996s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785837517 (560h44m56.656920552s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.785879817 (560h44m56.656962862s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786207791 (560h44m56.657290826s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786482865 (560h44m56.6575659s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.786771245 (560h44m56.65785428s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787031961 (560h44m56.658114996s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.787670256 (560h44m56.658753291s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788099514 (560h44m56.659182549s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788476843 (560h44m56.659559878s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.788785009 (560h44m56.659868054s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789180001 (560h44m56.660263036s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.789713827 (560h44m56.660796863s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.790063784 (560h44m56.661146819s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:56.791131708 (560h44m56.662214743s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Uploading piece...
Dec 29 00:13:58.502036914 (560h44m58.373119949s) [upload=491623]: Next returned nil segment
Dec 29 00:13:58.502054107 (560h44m58.373137152s) [upload=491623]: Waiting for error group managing segments...
Dec 29 00:13:58.789868406 (560h44m58.660951441s) [noise=true , address=145.79.227.26:4653   , piece_id=6ZOSBS3AKNLLYYXIFP7X7T3VBOUB2LJOOMP7DLTLHL2ARCNLXISQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.795195310 (560h44m58.666278355s) [noise=true , address=145.79.227.3:4149    , piece_id=SQCIY4XWMT5TUWVRUAQY2R27SIQMW25FSRKQAIJCFOJKGSKECDZA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.796611455 (560h44m58.667694491s) [noise=true , address=145.79.227.38:4978   , piece_id=556NT2ZDNR4MV6LPGUSLWROKX5L4EZTTWJZ5O42AP6S6LIPTCYGQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.800487035 (560h44m58.67157007s) [noise=true , address=145.79.227.42:4123   , piece_id=URZV5NE4W62W3PXR5LYNX22I666QTHPN52JHUGQ3VP3755W6AZMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.807920891 (560h44m58.679003936s) [noise=true , address=145.79.227.13:4804   , piece_id=5EBSWYDNF66K4HJMNWGBQPESWNLBEE3D5IDFGW6OX6HJD3RVQ4GQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.808728328 (560h44m58.679811363s) [noise=true , address=145.79.227.48:4301   , piece_id=EJZLEQNHU36RLPN6AEZUF3BVVMWOJJ4GULZZW457WN5JFJ57K6PQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.818637967 (560h44m58.689721002s) [noise=true , address=145.79.227.6:4258    , piece_id=IMQUXB4AUCK5WSL2QTFUOZ6LI6V2FKMSPEI4KFLS55MUOYY3Z4SA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:58.992291990 (560h44m58.863375025s) [noise=true , address=77.90.31.10:4518     , piece_id=2NYPG3LI26O4SHN7W6XTZWTSVBCWPQTMCBRNXY5FA3KGZQGNSV3Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.028644179 (560h44m58.899727214s) [noise=true , address=45.91.116.7:4009     , piece_id=EUW44ILO2EGAJ26BU3O67YNO7DUF4Y2FLSIF6GX6UKJHBB4XMVPQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.070650735 (560h44m58.94173377s) [noise=true , address=45.91.116.9:4264     , piece_id=YIKGA6QOEFENKJYBABWW2H6HYIZT5SJSJJZVBTSPV5PTMOZUCILA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.089366112 (560h44m58.960449158s) [noise=true , address=45.91.116.5:4299     , piece_id=C2UAXUGNGKSH3S5OEOV355OPBWNXLHEPYH3EJCG7RR2UYVIUWTMA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.094008442 (560h44m58.965091477s) [noise=true , address=45.91.116.10:4081    , piece_id=R5GULBXNILGRJNSAFYHCA4PQK6WCGRZ4AWR25WEPDSWSEEF5AQNA, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189151949 (560h44m59.060234984s) [noise=true , address=45.91.116.14:4191    , piece_id=3EY33YPGUBCAAPWQKDWPYQDS27OKUXWNA7XDBSRPFEBA2LFELF3A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: <nil>
Dec 29 00:13:59.189169192 (560h44m59.060252227s) [seg_pos={0 0}, upload=491623]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:13:59.189361147 (560h44m59.060444182s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF
Dec 29 00:13:59.189367199 (560h44m59.060450234s) [noise=true , address=145.79.227.44:4193   , piece_id=5JLRVM6WWSE3GPMOMSYVLZ7Z7J2U2HLYI4IFHQEMTRIVKM3GGJ6A, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189679433 (560h44m59.060762468s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: read tcp 188.166.225.170:53002->77.90.31.13:4627: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189694703 (560h44m59.060777738s) [noise=true , address=77.90.31.13:4627     , piece_id=ATLIC435WBNW7UQRNM2SPTZNKWWYE75UU4T3EUYLMJEFKJTVYYZQ, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189710252 (560h44m59.060793287s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: read tcp 188.166.225.170:49730->77.90.31.12:4589: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 29 00:13:59.189717145 (560h44m59.06080018s) [noise=true , address=77.90.31.12:4589     , piece_id=YMETGJ6WRUTYLJS3VZ5PZCFXZ4PKDFZAC5MLHIHJBW7UJZWZ6A2Q, seg_pos={0 0}, upload=491623]: Long tail cancellation due to optimal threshold
Dec 29 00:13:59.189730892 (560h44m59.060813927s) [seg_pos={0 0}, upload=491623]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:13:59.189791677 (560h44m59.060874712s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:13:59.189818598 (560h44m59.060901633s) [upload=491623]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:13:59.479474574 (560h44m59.350557609s) [upload=491623]: Done uploading segments...
Dec 29 00:13:59.479497958 (560h44m59.350581003s) [upload=491623]: Upload finished. err: <nil>
Duration: 30.00220128s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [...............................................X...X........]
duration: 
15.548674492s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:13:34 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 15.548701976s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.X.....X.....X......X...X.............................X...X.]
duration: 
5.768950043s
trace: 
https://jaeger.dev.storj.io/trace/29477b76de34a0b8
FAIL (2025-12-29 00:13:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:12:29.934430939 (560h43m29.805513974s) [upload=491603]: Starting upload
Dec 29 00:12:29.934489300 (560h43m29.805572345s) [upload=491603]: Done starting upload
Dec 29 00:12:29.934648503 (560h43m29.805731548s) [upload=491603]: Uploading segments...
Dec 29 00:12:29.934875425 (560h43m29.80595847s) [upload=491603]: Got next segment. Inline: false
Dec 29 00:12:29.934950267 (560h43m29.806033302s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:12:29.934957552 (560h43m29.806040587s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:12:30.178564544 (560h43m30.04964759s) [seg_pos={0 0}, upload=491603]: Begin upload segment...
Dec 29 00:12:30.178609630 (560h43m30.049692665s) [seg_pos={0 0}, upload=491603]: Done begin upload segment.
Dec 29 00:12:30.178638345 (560h43m30.04972139s) [noise=true , address=77.90.31.14:4664     , piece_id=VQMFEZSIOAJTYPV4ILI4IFJZ4RFZRKHJARYXVSPEZLRBMQPCZNHA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.178682118 (560h43m30.049765163s) [noise=true , address=145.79.227.41:4087   , piece_id=6KD3GB6HRPZ6COFEEIRJKGL2DKL2WS6XCFSCCG3HTQ55FH4VTAVQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.179439250 (560h43m30.050522285s) [noise=true , address=45.91.116.5:4298     , piece_id=VXF4GUGAWECU7DHGXBVR5NN2DJCI3YHMPEP3UG43W3TJYD7O57XQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.179804385 (560h43m30.05088743s) [noise=true , address=145.79.227.38:4979   , piece_id=YTC3YRR4HU2VNLEIMMCBQV7NSLUMGLCGP2O3XH73EJEY3IA3MYYQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.180123153 (560h43m30.051206188s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.180420919 (560h43m30.051503964s) [noise=true , address=145.79.227.32:4835   , piece_id=NEFXMRNLXLDXR6CZSFN4EK34Y4XAJTLZT465F5OQ5SJQN7GE3Q5Q, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181026192 (560h43m30.052109237s) [noise=true , address=145.79.227.23:4547   , piece_id=THM52EAJM3CXPI574HEPUJONYOABCJNGVDUMXTVRXWNJS4MC4WSA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181210363 (560h43m30.052293398s) [noise=true , address=45.91.116.9:4261     , piece_id=MXZDWELLHERV6DK4LAPWU24XFCG3PVPBGPTZRWR454B63TKAYSEQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181602639 (560h43m30.052685684s) [noise=true , address=145.79.227.52:4446   , piece_id=K4T7NOW2BG6A52J2PIOXUU4C3FSWM3U67CPPBURSLSBK62TOO3PA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181620113 (560h43m30.052703158s) [noise=true , address=145.79.227.50:4375   , piece_id=5BXJTOIU4HZBONZLJVTP42SLW66WKMKFYFAZM5RHANN2XYED4MTQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181938730 (560h43m30.053021765s) [noise=true , address=145.79.227.11:4732   , piece_id=SK6GQPKRHAGSLYZ57234TPWYVMX3WIBXCIXU3JERKWE7QARSQFZQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182209936 (560h43m30.053292971s) [noise=true , address=145.79.227.3:4151    , piece_id=CCENZOKHXVML2O6XOXEQGKXS2IYHU2UHU7FLN4HYYVU3NROKX6HQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182305888 (560h43m30.053388933s) [noise=true , address=45.91.116.10:4082    , piece_id=O5Z77TVPCEC72IKMWAGDZLDW2ZL6WYQOUUCWXQL3IDX3GJIIRIOQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182491252 (560h43m30.053574297s) [noise=true , address=45.91.116.7:4011     , piece_id=RB2S4PZTZRDRFB24H6KFXHTPFK2W7UY3HLM66LXGCIACGGBPRP7A, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182784530 (560h43m30.053867565s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.183068741 (560h43m30.054151776s) [noise=true , address=145.79.227.47:4265   , piece_id=OC4YNMFV24O6ZCWJ3BDX56SQRAXJ3BRAMMGRSD64JXZ4NZOQKVNA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:31.888560119 (560h43m31.759643154s) [upload=491603]: Next returned nil segment
Dec 29 00:12:31.888573986 (560h43m31.759657021s) [upload=491603]: Waiting for error group managing segments...
Dec 29 00:12:32.170065044 (560h43m32.041148089s) [noise=true , address=145.79.227.50:4375   , piece_id=5BXJTOIU4HZBONZLJVTP42SLW66WKMKFYFAZM5RHANN2XYED4MTQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.182788382 (560h43m32.053871417s) [noise=true , address=145.79.227.38:4979   , piece_id=YTC3YRR4HU2VNLEIMMCBQV7NSLUMGLCGP2O3XH73EJEY3IA3MYYQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.185480608 (560h43m32.056563643s) [noise=true , address=145.79.227.3:4151    , piece_id=CCENZOKHXVML2O6XOXEQGKXS2IYHU2UHU7FLN4HYYVU3NROKX6HQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.186206620 (560h43m32.057289655s) [noise=true , address=145.79.227.41:4087   , piece_id=6KD3GB6HRPZ6COFEEIRJKGL2DKL2WS6XCFSCCG3HTQ55FH4VTAVQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.201309659 (560h43m32.072392694s) [noise=true , address=145.79.227.52:4446   , piece_id=K4T7NOW2BG6A52J2PIOXUU4C3FSWM3U67CPPBURSLSBK62TOO3PA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.206502326 (560h43m32.077585361s) [noise=true , address=145.79.227.23:4547   , piece_id=THM52EAJM3CXPI574HEPUJONYOABCJNGVDUMXTVRXWNJS4MC4WSA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.209887873 (560h43m32.080970908s) [noise=true , address=145.79.227.32:4835   , piece_id=NEFXMRNLXLDXR6CZSFN4EK34Y4XAJTLZT465F5OQ5SJQN7GE3Q5Q, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.212673006 (560h43m32.083756051s) [noise=true , address=145.79.227.11:4732   , piece_id=SK6GQPKRHAGSLYZ57234TPWYVMX3WIBXCIXU3JERKWE7QARSQFZQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.215966166 (560h43m32.087049201s) [noise=true , address=145.79.227.47:4265   , piece_id=OC4YNMFV24O6ZCWJ3BDX56SQRAXJ3BRAMMGRSD64JXZ4NZOQKVNA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.375399070 (560h43m32.246482105s) [noise=true , address=77.90.31.14:4664     , piece_id=VQMFEZSIOAJTYPV4ILI4IFJZ4RFZRKHJARYXVSPEZLRBMQPCZNHA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.435386925 (560h43m32.30646996s) [noise=true , address=45.91.116.10:4082    , piece_id=O5Z77TVPCEC72IKMWAGDZLDW2ZL6WYQOUUCWXQL3IDX3GJIIRIOQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.461154801 (560h43m32.332237846s) [noise=true , address=45.91.116.9:4261     , piece_id=MXZDWELLHERV6DK4LAPWU24XFCG3PVPBGPTZRWR454B63TKAYSEQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478222260 (560h43m32.349305295s) [noise=true , address=45.91.116.5:4298     , piece_id=VXF4GUGAWECU7DHGXBVR5NN2DJCI3YHMPEP3UG43W3TJYD7O57XQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478243019 (560h43m32.349326054s) [seg_pos={0 0}, upload=491603]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:12:32.478424414 (560h43m32.349507449s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 00:12:32.478433222 (560h43m32.349516257s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Long tail cancellation due to optimal threshold
Dec 29 00:12:32.478653461 (560h43m32.349736506s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 29 00:12:32.478658801 (560h43m32.349741836s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Long tail cancellation due to optimal threshold
Dec 29 00:12:32.478757058 (560h43m32.349840093s) [noise=true , address=45.91.116.7:4011     , piece_id=RB2S4PZTZRDRFB24H6KFXHTPFK2W7UY3HLM66LXGCIACGGBPRP7A, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478794710 (560h43m32.349877745s) [seg_pos={0 0}, upload=491603]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 29 00:12:32.478877267 (560h43m32.349960302s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:12:32.478882166 (560h43m32.349965201s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:12:32.816007069 (560h43m32.687090104s) [upload=491603]: Done uploading segments...
Dec 29 00:12:32.816151634 (560h43m32.687234669s) [upload=491603]: Upload finished. err: <nil>
Duration: 5.768964911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [X...............................................X...X.......]
duration: 
46.73889837s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:13:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 46.73892448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..X.....X.....X......X...X.............................X...X]
duration: 
30.002694231s
trace: 
https://jaeger.dev.storj.io/trace/29477b76de34a0b8
FAIL (2025-12-29 00:12:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 29 00:12:29.934430939 (560h43m29.805513974s) [upload=491603]: Starting upload
Dec 29 00:12:29.934489300 (560h43m29.805572345s) [upload=491603]: Done starting upload
Dec 29 00:12:29.934648503 (560h43m29.805731548s) [upload=491603]: Uploading segments...
Dec 29 00:12:29.934875425 (560h43m29.80595847s) [upload=491603]: Got next segment. Inline: false
Dec 29 00:12:29.934950267 (560h43m29.806033302s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:12:29.934957552 (560h43m29.806040587s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:12:30.178564544 (560h43m30.04964759s) [seg_pos={0 0}, upload=491603]: Begin upload segment...
Dec 29 00:12:30.178609630 (560h43m30.049692665s) [seg_pos={0 0}, upload=491603]: Done begin upload segment.
Dec 29 00:12:30.178638345 (560h43m30.04972139s) [noise=true , address=77.90.31.14:4664     , piece_id=VQMFEZSIOAJTYPV4ILI4IFJZ4RFZRKHJARYXVSPEZLRBMQPCZNHA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.178682118 (560h43m30.049765163s) [noise=true , address=145.79.227.41:4087   , piece_id=6KD3GB6HRPZ6COFEEIRJKGL2DKL2WS6XCFSCCG3HTQ55FH4VTAVQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.179439250 (560h43m30.050522285s) [noise=true , address=45.91.116.5:4298     , piece_id=VXF4GUGAWECU7DHGXBVR5NN2DJCI3YHMPEP3UG43W3TJYD7O57XQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.179804385 (560h43m30.05088743s) [noise=true , address=145.79.227.38:4979   , piece_id=YTC3YRR4HU2VNLEIMMCBQV7NSLUMGLCGP2O3XH73EJEY3IA3MYYQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.180123153 (560h43m30.051206188s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.180420919 (560h43m30.051503964s) [noise=true , address=145.79.227.32:4835   , piece_id=NEFXMRNLXLDXR6CZSFN4EK34Y4XAJTLZT465F5OQ5SJQN7GE3Q5Q, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181026192 (560h43m30.052109237s) [noise=true , address=145.79.227.23:4547   , piece_id=THM52EAJM3CXPI574HEPUJONYOABCJNGVDUMXTVRXWNJS4MC4WSA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181210363 (560h43m30.052293398s) [noise=true , address=45.91.116.9:4261     , piece_id=MXZDWELLHERV6DK4LAPWU24XFCG3PVPBGPTZRWR454B63TKAYSEQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181602639 (560h43m30.052685684s) [noise=true , address=145.79.227.52:4446   , piece_id=K4T7NOW2BG6A52J2PIOXUU4C3FSWM3U67CPPBURSLSBK62TOO3PA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181620113 (560h43m30.052703158s) [noise=true , address=145.79.227.50:4375   , piece_id=5BXJTOIU4HZBONZLJVTP42SLW66WKMKFYFAZM5RHANN2XYED4MTQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.181938730 (560h43m30.053021765s) [noise=true , address=145.79.227.11:4732   , piece_id=SK6GQPKRHAGSLYZ57234TPWYVMX3WIBXCIXU3JERKWE7QARSQFZQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182209936 (560h43m30.053292971s) [noise=true , address=145.79.227.3:4151    , piece_id=CCENZOKHXVML2O6XOXEQGKXS2IYHU2UHU7FLN4HYYVU3NROKX6HQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182305888 (560h43m30.053388933s) [noise=true , address=45.91.116.10:4082    , piece_id=O5Z77TVPCEC72IKMWAGDZLDW2ZL6WYQOUUCWXQL3IDX3GJIIRIOQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182491252 (560h43m30.053574297s) [noise=true , address=45.91.116.7:4011     , piece_id=RB2S4PZTZRDRFB24H6KFXHTPFK2W7UY3HLM66LXGCIACGGBPRP7A, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.182784530 (560h43m30.053867565s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:30.183068741 (560h43m30.054151776s) [noise=true , address=145.79.227.47:4265   , piece_id=OC4YNMFV24O6ZCWJ3BDX56SQRAXJ3BRAMMGRSD64JXZ4NZOQKVNA, seg_pos={0 0}, upload=491603]: Uploading piece...
Dec 29 00:12:31.888560119 (560h43m31.759643154s) [upload=491603]: Next returned nil segment
Dec 29 00:12:31.888573986 (560h43m31.759657021s) [upload=491603]: Waiting for error group managing segments...
Dec 29 00:12:32.170065044 (560h43m32.041148089s) [noise=true , address=145.79.227.50:4375   , piece_id=5BXJTOIU4HZBONZLJVTP42SLW66WKMKFYFAZM5RHANN2XYED4MTQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.182788382 (560h43m32.053871417s) [noise=true , address=145.79.227.38:4979   , piece_id=YTC3YRR4HU2VNLEIMMCBQV7NSLUMGLCGP2O3XH73EJEY3IA3MYYQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.185480608 (560h43m32.056563643s) [noise=true , address=145.79.227.3:4151    , piece_id=CCENZOKHXVML2O6XOXEQGKXS2IYHU2UHU7FLN4HYYVU3NROKX6HQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.186206620 (560h43m32.057289655s) [noise=true , address=145.79.227.41:4087   , piece_id=6KD3GB6HRPZ6COFEEIRJKGL2DKL2WS6XCFSCCG3HTQ55FH4VTAVQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.201309659 (560h43m32.072392694s) [noise=true , address=145.79.227.52:4446   , piece_id=K4T7NOW2BG6A52J2PIOXUU4C3FSWM3U67CPPBURSLSBK62TOO3PA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.206502326 (560h43m32.077585361s) [noise=true , address=145.79.227.23:4547   , piece_id=THM52EAJM3CXPI574HEPUJONYOABCJNGVDUMXTVRXWNJS4MC4WSA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.209887873 (560h43m32.080970908s) [noise=true , address=145.79.227.32:4835   , piece_id=NEFXMRNLXLDXR6CZSFN4EK34Y4XAJTLZT465F5OQ5SJQN7GE3Q5Q, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.212673006 (560h43m32.083756051s) [noise=true , address=145.79.227.11:4732   , piece_id=SK6GQPKRHAGSLYZ57234TPWYVMX3WIBXCIXU3JERKWE7QARSQFZQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.215966166 (560h43m32.087049201s) [noise=true , address=145.79.227.47:4265   , piece_id=OC4YNMFV24O6ZCWJ3BDX56SQRAXJ3BRAMMGRSD64JXZ4NZOQKVNA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.375399070 (560h43m32.246482105s) [noise=true , address=77.90.31.14:4664     , piece_id=VQMFEZSIOAJTYPV4ILI4IFJZ4RFZRKHJARYXVSPEZLRBMQPCZNHA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.435386925 (560h43m32.30646996s) [noise=true , address=45.91.116.10:4082    , piece_id=O5Z77TVPCEC72IKMWAGDZLDW2ZL6WYQOUUCWXQL3IDX3GJIIRIOQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.461154801 (560h43m32.332237846s) [noise=true , address=45.91.116.9:4261     , piece_id=MXZDWELLHERV6DK4LAPWU24XFCG3PVPBGPTZRWR454B63TKAYSEQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478222260 (560h43m32.349305295s) [noise=true , address=45.91.116.5:4298     , piece_id=VXF4GUGAWECU7DHGXBVR5NN2DJCI3YHMPEP3UG43W3TJYD7O57XQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478243019 (560h43m32.349326054s) [seg_pos={0 0}, upload=491603]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:12:32.478424414 (560h43m32.349507449s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125fn3wMJUNjsKTgsssLJkZAobCbaAS9jC2VyxXxbUA2ei4Gjni): protocol: expected piece hash; context canceled; EOF
Dec 29 00:12:32.478433222 (560h43m32.349516257s) [noise=true , address=145.79.227.29:4941   , piece_id=PDJZZGLMSN4A5ANOFBMSHOYALG7OYYMODRGM5G4RJPSOM4L2DWKQ, seg_pos={0 0}, upload=491603]: Long tail cancellation due to optimal threshold
Dec 29 00:12:32.478653461 (560h43m32.349736506s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 29 00:12:32.478658801 (560h43m32.349741836s) [noise=true , address=45.91.116.8:4047     , piece_id=6Q2N4GXMLARVD7EF5BWTTD3S6EGJXNDBMADS4Z7HCHU7LQNUPCNA, seg_pos={0 0}, upload=491603]: Long tail cancellation due to optimal threshold
Dec 29 00:12:32.478757058 (560h43m32.349840093s) [noise=true , address=45.91.116.7:4011     , piece_id=RB2S4PZTZRDRFB24H6KFXHTPFK2W7UY3HLM66LXGCIACGGBPRP7A, seg_pos={0 0}, upload=491603]: Done uploading piece. err: <nil>
Dec 29 00:12:32.478794710 (560h43m32.349877745s) [seg_pos={0 0}, upload=491603]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 29 00:12:32.478877267 (560h43m32.349960302s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:12:32.478882166 (560h43m32.349965201s) [upload=491603]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:12:32.816007069 (560h43m32.687090104s) [upload=491603]: Done uploading segments...
Dec 29 00:12:32.816151634 (560h43m32.687234669s) [upload=491603]: Upload finished. err: <nil>
Duration: 30.00271948s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.X...............................................X...X......]
duration: 
15.404980045s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:12:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 15.405012638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [...X.....X.....X......X...X.............................X...]
duration: 
6.297109123s
trace: 
https://jaeger.dev.storj.io/trace/1d4a343f61401e70
FAIL (2025-12-29 00:11:35 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
:09:27.869952873 (560h40m27.741035908s) [upload=491555]: Starting upload
Dec 29 00:09:27.870040369 (560h40m27.741123414s) [upload=491555]: Done starting upload
Dec 29 00:09:27.870179464 (560h40m27.741262519s) [upload=491555]: Uploading segments...
Dec 29 00:09:27.870306316 (560h40m27.741389371s) [upload=491555]: Got next segment. Inline: false
Dec 29 00:09:27.870355480 (560h40m27.741438535s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:09:27.870383552 (560h40m27.741466607s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:09:28.116097046 (560h40m27.987180081s) [seg_pos={0 0}, upload=491555]: Begin upload segment...
Dec 29 00:09:28.116156770 (560h40m27.987239815s) [seg_pos={0 0}, upload=491555]: Done begin upload segment.
Dec 29 00:09:28.116195053 (560h40m27.987278098s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.116839099 (560h40m27.987922144s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.117528552 (560h40m27.988611598s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118108366 (560h40m27.989191411s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118586777 (560h40m27.989669822s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119025913 (560h40m27.990108958s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119440883 (560h40m27.990523928s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119885430 (560h40m27.990968465s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.121433417 (560h40m27.992516472s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123097756 (560h40m27.994180791s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123861831 (560h40m27.994944876s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.124632759 (560h40m27.995715794s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.126502069 (560h40m27.997585114s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.128576139 (560h40m27.999659184s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.129102481 (560h40m28.000185526s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.130895795 (560h40m28.00197883s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:29.843367504 (560h40m29.714450539s) [upload=491555]: Next returned nil segment
Dec 29 00:09:29.843385218 (560h40m29.714468283s) [upload=491555]: Waiting for error group managing segments...
Dec 29 00:09:30.127111674 (560h40m29.998194709s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.128417710 (560h40m29.999500745s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.140425976 (560h40m30.011509021s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.147320685 (560h40m30.01840372s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.148896696 (560h40m30.019979731s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.154020161 (560h40m30.025103206s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.184289699 (560h40m30.055372734s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.193493144 (560h40m30.064576189s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.244340955 (560h40m30.11542399s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.384632981 (560h40m30.255716026s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395196796 (560h40m30.266279831s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395954068 (560h40m30.267037103s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421727925 (560h40m30.29281097s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421753704 (560h40m30.292836749s) [seg_pos={0 0}, upload=491555]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:09:30.422007768 (560h40m30.293090803s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422015022 (560h40m30.293098067s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422244849 (560h40m30.293327894s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422266351 (560h40m30.293349396s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422339049 (560h40m30.293422084s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection
Dec 29 00:09:30.422347815 (560h40m30.29343085s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422367884 (560h40m30.293450919s) [seg_pos={0 0}, upload=491555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:09:30.422443778 (560h40m30.293526813s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:09:30.422449118 (560h40m30.293532163s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:09:30.709063503 (560h40m30.580146548s) [upload=491555]: Done uploading segments...
Dec 29 00:09:30.709088030 (560h40m30.580171075s) [upload=491555]: Upload finished. err: <nil>
Duration: 6.297146785s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [..X...............................................X...X.....]
duration: 
31.192693477s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:11:32 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 31.192724334s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
6 out of 60 is failed (threshold=3) [....X.....X.....X......X...X.............................X..]
duration: 
5.980960287s
trace: 
https://jaeger.dev.storj.io/trace/1d4a343f61401e70
FAIL (2025-12-29 00:11:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
:09:27.869952873 (560h40m27.741035908s) [upload=491555]: Starting upload
Dec 29 00:09:27.870040369 (560h40m27.741123414s) [upload=491555]: Done starting upload
Dec 29 00:09:27.870179464 (560h40m27.741262519s) [upload=491555]: Uploading segments...
Dec 29 00:09:27.870306316 (560h40m27.741389371s) [upload=491555]: Got next segment. Inline: false
Dec 29 00:09:27.870355480 (560h40m27.741438535s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:09:27.870383552 (560h40m27.741466607s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:09:28.116097046 (560h40m27.987180081s) [seg_pos={0 0}, upload=491555]: Begin upload segment...
Dec 29 00:09:28.116156770 (560h40m27.987239815s) [seg_pos={0 0}, upload=491555]: Done begin upload segment.
Dec 29 00:09:28.116195053 (560h40m27.987278098s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.116839099 (560h40m27.987922144s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.117528552 (560h40m27.988611598s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118108366 (560h40m27.989191411s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118586777 (560h40m27.989669822s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119025913 (560h40m27.990108958s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119440883 (560h40m27.990523928s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119885430 (560h40m27.990968465s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.121433417 (560h40m27.992516472s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123097756 (560h40m27.994180791s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123861831 (560h40m27.994944876s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.124632759 (560h40m27.995715794s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.126502069 (560h40m27.997585114s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.128576139 (560h40m27.999659184s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.129102481 (560h40m28.000185526s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.130895795 (560h40m28.00197883s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:29.843367504 (560h40m29.714450539s) [upload=491555]: Next returned nil segment
Dec 29 00:09:29.843385218 (560h40m29.714468283s) [upload=491555]: Waiting for error group managing segments...
Dec 29 00:09:30.127111674 (560h40m29.998194709s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.128417710 (560h40m29.999500745s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.140425976 (560h40m30.011509021s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.147320685 (560h40m30.01840372s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.148896696 (560h40m30.019979731s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.154020161 (560h40m30.025103206s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.184289699 (560h40m30.055372734s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.193493144 (560h40m30.064576189s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.244340955 (560h40m30.11542399s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.384632981 (560h40m30.255716026s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395196796 (560h40m30.266279831s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395954068 (560h40m30.267037103s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421727925 (560h40m30.29281097s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421753704 (560h40m30.292836749s) [seg_pos={0 0}, upload=491555]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:09:30.422007768 (560h40m30.293090803s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422015022 (560h40m30.293098067s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422244849 (560h40m30.293327894s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422266351 (560h40m30.293349396s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422339049 (560h40m30.293422084s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection
Dec 29 00:09:30.422347815 (560h40m30.29343085s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422367884 (560h40m30.293450919s) [seg_pos={0 0}, upload=491555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:09:30.422443778 (560h40m30.293526813s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:09:30.422449118 (560h40m30.293532163s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:09:30.709063503 (560h40m30.580146548s) [upload=491555]: Done uploading segments...
Dec 29 00:09:30.709088030 (560h40m30.580171075s) [upload=491555]: Upload finished. err: <nil>
Duration: 5.98098797s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [...X...............................................X...X....]
duration: 
15.42363992s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:10:59 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 15.423667933s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [X....X.....X.....X......X...X.............................X.]
duration: 
9.08155864s
trace: 
https://jaeger.dev.storj.io/trace/1d4a343f61401e70
FAIL (2025-12-29 00:10:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
:09:27.869952873 (560h40m27.741035908s) [upload=491555]: Starting upload
Dec 29 00:09:27.870040369 (560h40m27.741123414s) [upload=491555]: Done starting upload
Dec 29 00:09:27.870179464 (560h40m27.741262519s) [upload=491555]: Uploading segments...
Dec 29 00:09:27.870306316 (560h40m27.741389371s) [upload=491555]: Got next segment. Inline: false
Dec 29 00:09:27.870355480 (560h40m27.741438535s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:09:27.870383552 (560h40m27.741466607s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:09:28.116097046 (560h40m27.987180081s) [seg_pos={0 0}, upload=491555]: Begin upload segment...
Dec 29 00:09:28.116156770 (560h40m27.987239815s) [seg_pos={0 0}, upload=491555]: Done begin upload segment.
Dec 29 00:09:28.116195053 (560h40m27.987278098s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.116839099 (560h40m27.987922144s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.117528552 (560h40m27.988611598s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118108366 (560h40m27.989191411s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118586777 (560h40m27.989669822s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119025913 (560h40m27.990108958s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119440883 (560h40m27.990523928s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119885430 (560h40m27.990968465s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.121433417 (560h40m27.992516472s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123097756 (560h40m27.994180791s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123861831 (560h40m27.994944876s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.124632759 (560h40m27.995715794s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.126502069 (560h40m27.997585114s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.128576139 (560h40m27.999659184s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.129102481 (560h40m28.000185526s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.130895795 (560h40m28.00197883s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:29.843367504 (560h40m29.714450539s) [upload=491555]: Next returned nil segment
Dec 29 00:09:29.843385218 (560h40m29.714468283s) [upload=491555]: Waiting for error group managing segments...
Dec 29 00:09:30.127111674 (560h40m29.998194709s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.128417710 (560h40m29.999500745s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.140425976 (560h40m30.011509021s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.147320685 (560h40m30.01840372s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.148896696 (560h40m30.019979731s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.154020161 (560h40m30.025103206s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.184289699 (560h40m30.055372734s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.193493144 (560h40m30.064576189s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.244340955 (560h40m30.11542399s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.384632981 (560h40m30.255716026s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395196796 (560h40m30.266279831s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395954068 (560h40m30.267037103s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421727925 (560h40m30.29281097s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421753704 (560h40m30.292836749s) [seg_pos={0 0}, upload=491555]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:09:30.422007768 (560h40m30.293090803s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422015022 (560h40m30.293098067s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422244849 (560h40m30.293327894s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422266351 (560h40m30.293349396s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422339049 (560h40m30.293422084s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection
Dec 29 00:09:30.422347815 (560h40m30.29343085s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422367884 (560h40m30.293450919s) [seg_pos={0 0}, upload=491555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:09:30.422443778 (560h40m30.293526813s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:09:30.422449118 (560h40m30.293532163s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:09:30.709063503 (560h40m30.580146548s) [upload=491555]: Done uploading segments...
Dec 29 00:09:30.709088030 (560h40m30.580171075s) [upload=491555]: Upload finished. err: <nil>
Duration: 9.081586692s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [....X...............................................X...X...]
duration: 
11.491892284s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:10:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 11.491920206s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.X....X.....X.....X......X...X.............................X]
duration: 
30.003279473s
trace: 
https://jaeger.dev.storj.io/trace/1d4a343f61401e70
FAIL (2025-12-29 00:09:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
:09:27.869952873 (560h40m27.741035908s) [upload=491555]: Starting upload
Dec 29 00:09:27.870040369 (560h40m27.741123414s) [upload=491555]: Done starting upload
Dec 29 00:09:27.870179464 (560h40m27.741262519s) [upload=491555]: Uploading segments...
Dec 29 00:09:27.870306316 (560h40m27.741389371s) [upload=491555]: Got next segment. Inline: false
Dec 29 00:09:27.870355480 (560h40m27.741438535s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 00:09:27.870383552 (560h40m27.741466607s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 00:09:28.116097046 (560h40m27.987180081s) [seg_pos={0 0}, upload=491555]: Begin upload segment...
Dec 29 00:09:28.116156770 (560h40m27.987239815s) [seg_pos={0 0}, upload=491555]: Done begin upload segment.
Dec 29 00:09:28.116195053 (560h40m27.987278098s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.116839099 (560h40m27.987922144s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.117528552 (560h40m27.988611598s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118108366 (560h40m27.989191411s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.118586777 (560h40m27.989669822s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119025913 (560h40m27.990108958s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119440883 (560h40m27.990523928s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.119885430 (560h40m27.990968465s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.121433417 (560h40m27.992516472s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123097756 (560h40m27.994180791s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.123861831 (560h40m27.994944876s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.124632759 (560h40m27.995715794s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.126502069 (560h40m27.997585114s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.128576139 (560h40m27.999659184s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.129102481 (560h40m28.000185526s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:28.130895795 (560h40m28.00197883s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Uploading piece...
Dec 29 00:09:29.843367504 (560h40m29.714450539s) [upload=491555]: Next returned nil segment
Dec 29 00:09:29.843385218 (560h40m29.714468283s) [upload=491555]: Waiting for error group managing segments...
Dec 29 00:09:30.127111674 (560h40m29.998194709s) [noise=true , address=145.79.227.51:4409   , piece_id=ADAW5ZYX2GQOQUBPQE5EJDWRPYHHUFDZLCYPTYFT6YQSDKW4NRGA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.128417710 (560h40m29.999500745s) [noise=true , address=145.79.227.6:4257    , piece_id=FPL5TNXFTZTWH4RJSQVFIHE66JI5MMUMVEGFT6WXES2TQI7MQ3EQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.140425976 (560h40m30.011509021s) [noise=true , address=145.79.227.11:4730   , piece_id=SKV37E7UAEMRPA2EEK2UNVAK3QPYUZSF732SWCAJOZUFVUE3YFEQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.147320685 (560h40m30.01840372s) [noise=true , address=145.79.227.45:4230   , piece_id=ZU4WK5H7MR6XRVD3Z3QQ3YFDMWY25BD54JLPIZRDLYJ5EVJ4FX2Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.148896696 (560h40m30.019979731s) [noise=true , address=145.79.227.15:4874   , piece_id=AVX2QNRKRL5Y4Q2FSAODYX35AH2ZOONYAID2DHSLPBWSR5I2JNWA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.154020161 (560h40m30.025103206s) [noise=true , address=145.79.227.39:4013   , piece_id=BS33WXEHX76KGOGTPNQEPTN65IW53N26AOMIYHAJT2R6W2RFJVTA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.184289699 (560h40m30.055372734s) [noise=true , address=145.79.227.47:4267   , piece_id=GPMJQ7HK7N7F3FC62NTGNTBQSS6LPEI5JIIJ5O2EWMABLDUJ6IYA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.193493144 (560h40m30.064576189s) [noise=true , address=145.79.227.13:4802   , piece_id=D2GT2G5WHTHHMZHYBIYMN4TMQ3PVPBGF4HOF5RXTJ7HEMYDTVY7Q, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.244340955 (560h40m30.11542399s) [noise=true , address=145.79.227.2:4114    , piece_id=2IGGHF5ZOQCCDU73CAPGN6PQ6VIRX5SOROSVDU73ALTATBA62WVQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.384632981 (560h40m30.255716026s) [noise=true , address=77.90.31.17:4770     , piece_id=C4DPRPPEC5IS4MO3JL5CUJ6NVGQ5GGSNP5I22QE6FGCVY2F3UQOA, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395196796 (560h40m30.266279831s) [noise=true , address=77.90.31.11:4556     , piece_id=DWKHDOSTS3TNMTMXLJGTKJF77WL6NLTWNNMB755ZQSQLWCASLOJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.395954068 (560h40m30.267037103s) [noise=true , address=77.90.31.21:4877     , piece_id=BVTZVOGKFUV62Q4PNXOBRNHN5CSWCM6AWZET3NNLI2OFRR5GRRJQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421727925 (560h40m30.29281097s) [noise=true , address=45.91.116.12:4225    , piece_id=7IHUQOJLZSVUB2W7KKIYTJISYLWMAQLZ4KAKH7IILHF2UJIKR7IQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: <nil>
Dec 29 00:09:30.421753704 (560h40m30.292836749s) [seg_pos={0 0}, upload=491555]: Segment reached optimal threshold of 13 pieces.
Dec 29 00:09:30.422007768 (560h40m30.293090803s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422015022 (560h40m30.293098067s) [noise=true , address=145.79.227.35:4481   , piece_id=FY5HIZBJQ5MRLFWHP2FRU23Q4QW6TLHX3UDS23UT2O5IYOWOYFGQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422244849 (560h40m30.293327894s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF
Dec 29 00:09:30.422266351 (560h40m30.293349396s) [noise=true , address=77.90.31.10:4517     , piece_id=XG6LCYNK5VF4RVWRYTQSRDV3EC567XDAB5VOWIOYI6RE7FDE4D2A, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422339049 (560h40m30.293422084s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp): protocol: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36626->145.79.227.18:4007: use of closed network connection
Dec 29 00:09:30.422347815 (560h40m30.29343085s) [noise=true , address=145.79.227.18:4007   , piece_id=GPTHMKLJXD6KSFKDSFFD3JQ7AEOXEP75FPZKBDRG77BY5IR64PPQ, seg_pos={0 0}, upload=491555]: Long tail cancellation due to optimal threshold
Dec 29 00:09:30.422367884 (560h40m30.293450919s) [seg_pos={0 0}, upload=491555]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 00:09:30.422443778 (560h40m30.293526813s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:09:30.422449118 (560h40m30.293532163s) [upload=491555]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:09:30.709063503 (560h40m30.580146548s) [upload=491555]: Done uploading segments...
Dec 29 00:09:30.709088030 (560h40m30.580171075s) [upload=491555]: Upload finished. err: <nil>
Duration: 30.003301866s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.....X...............................................X...X..]
duration: 
15.393946743s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:09:55 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 15.39396099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [......X...............................................X...X.]
duration: 
28.379406789s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:09:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 28.379424322s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
3 out of 60 is failed (threshold=3) [.......X...............................................X...X]
duration: 
1m0.003962816s
trace: 
https://jaeger.dev.storj.io/trace/4f67d6c3b1be911b
FAIL (2025-12-29 00:09:11 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
one begin upload segment.
Dec 29 00:08:12.297557448 (560h39m12.168640483s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.297708887 (560h39m12.168791932s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298357431 (560h39m12.169440466s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298822187 (560h39m12.169905222s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.298946423 (560h39m12.170029458s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299495348 (560h39m12.170578393s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.299822190 (560h39m12.170905225s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300117633 (560h39m12.171200668s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300163731 (560h39m12.171246766s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300565144 (560h39m12.171648189s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300891937 (560h39m12.171974982s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.300912596 (560h39m12.171995641s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301388803 (560h39m12.172471838s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.301573615 (560h39m12.17265666s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302186202 (560h39m12.173269237s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.302738824 (560h39m12.173821869s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303261810 (560h39m12.174344865s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:12.303710284 (560h39m12.174793319s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Uploading piece...
Dec 29 00:08:13.218615242 (560h39m13.089698287s) [noise=true , address=193.149.164.143:4702 , piece_id=YIIEGPMCCZKZQUFKSD4U5S2IMWK2A2HGHZH7XFDLJ2JDNQH5H3BQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:12cTKvv93eK129pGYE3ZXt81vByFHzwJjesrkYWFAjXvSK3dTC1, address:193.149.164.143:4702): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:13.602278430 (560h39m13.473361465s) [upload=491536]: Next returned nil segment
Dec 29 00:08:13.602292898 (560h39m13.473375933s) [upload=491536]: Waiting for error group managing segments...
Dec 29 00:08:13.930530389 (560h39m13.801613424s) [noise=true , address=64.49.13.139:4476    , piece_id=PCXYPVKNJAQVER33WPEKVSRUSOIPFX7JQHXAXGQ5ZZS3DCHHTBZQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:13.963144882 (560h39m13.834227917s) [noise=true , address=64.49.13.140:4510    , piece_id=62BEZRDVPCZQUYE2F33FM23APJWDTGKIJ5AJ56K5KITRAYPMK5YQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.274870980 (560h39m14.145954015s) [noise=true , address=145.79.227.14:4857   , piece_id=BDPLZLVBO7HIAECRPPA3B2C2VADMD72NLSS67KB23NQAHK2XKPCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.365760656 (560h39m14.236843701s) [noise=true , address=77.90.31.21:4895     , piece_id=GRJJNNMYTX6S4E6EVXRBMJNUPHZ767LQQ2SNBGXQ5NGQ377VGJVA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.522114575 (560h39m14.39319762s) [noise=true , address=5.231.73.2:4022      , piece_id=OHLIAQIUACULD5XQCKEBRIWHSERTRRIZL5INDRVK4QIN724YFYKA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.569478153 (560h39m14.440561198s) [noise=true , address=145.79.227.12:4771   , piece_id=N4VGO26WH4SRJCOWOCC37WXTJYRRU4S4Y5ZT74I54J4DENECIDEQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.570431819 (560h39m14.441514864s) [noise=true , address=5.231.73.4:4022      , piece_id=4ZHP4PYVRR3CBGNCW5AADSKVEAH7CR6ZMPJX7R6HUM7S7JK5S2NA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.699493561 (560h39m14.570576596s) [noise=true , address=77.90.31.17:4783     , piece_id=LVYTXRPXOBBVCJVGTY76U4RYLDHPGYLWWMJQ4QZQM2LZJC5RSYXQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.840061172 (560h39m14.711144227s) [noise=true , address=64.49.13.26:4948     , piece_id=YDAVJSVEFKG5P6SX35WNQKX54WGGVLFBSV6QQT7H634FN7KGRE5Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: ecclient: upload failed (node:1nVV4rAAZRd4DcLsakTTyPRNWck7AjBA6StKvsVLH2dCKVkwHB, address:64.49.13.26:4948): protocol: storage node overloaded, request limit: 2000
Dec 29 00:08:14.848741570 (560h39m14.719824605s) [noise=true , address=77.90.31.11:4586     , piece_id=IBVHAMXUPZESTBT4A7V63A25N5GOJYL534RMIHKQRK3CZABL6MDQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:14.988402886 (560h39m14.859485931s) [noise=true , address=64.49.13.154:4926    , piece_id=7G3VEWRTJ4OHCUAESXJWDYQIJOBOPOGYHXZMUVYAOSCNGIRHD37Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.039622174 (560h39m14.910705209s) [noise=true , address=145.79.227.30:4784   , piece_id=7B7YCYXKH77GLMSMC2MIGPJIIQXXPDDBXRPJHPFLWORH7NNF5SCA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.126270394 (560h39m14.997353439s) [noise=true , address=145.79.227.43:4178   , piece_id=OLZXZCQX3DXQ4GRSTFSVPIJ324HYKQAPBMS7E63DBPSLZMDWFFKQ, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.155355296 (560h39m15.026438331s) [noise=true , address=145.79.227.33:4896   , piece_id=4UMFNFYVHLPAWPUAPJMPB6I45KI7GW7LSWL2GSRBFO4SHVBQO62Q, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195186570 (560h39m15.066269615s) [noise=true , address=145.79.227.34:4930   , piece_id=QF73EKU5Q7FET2YJATF5LCGWINELVN2QTGRDEXLTDD7ZM2NWKN2A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: <nil>
Dec 29 00:08:15.195210045 (560h39m15.06629308s) [seg_pos={0 0}, upload=491536]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:08:15.195452026 (560h39m15.066535061s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XsPVNGYUhKnXkQRfVfjyBRzyvhShSdsrnzXtwe4Xtf5MaHMvH): protocol: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38176->145.79.227.17:4987: use of closed network connection
Dec 29 00:08:15.195459079 (560h39m15.066542124s) [noise=true , address=145.79.227.17:4987   , piece_id=A3M6DULNDQKIPUP2ONFOPVMS5P5Z62PVL3IZ5QPV5WUYK6MBCMHA, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195482404 (560h39m15.066565439s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rhkcipPBZNFScBjBhWdigzSMwENSmzhCucnC2iuyc9udpweE3): protocol: expected piece hash; context canceled; EOF
Dec 29 00:08:15.195498855 (560h39m15.06658189s) [noise=true , address=145.79.227.43:4180   , piece_id=NXAZB6NCQQIYFFVVSZ4X735TBNUTVVFRZVPQUTMGE6RXRCAGQX4A, seg_pos={0 0}, upload=491536]: Long tail cancellation due to optimal threshold
Dec 29 00:08:15.195525055 (560h39m15.0666081s) [seg_pos={0 0}, upload=491536]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:08:15.195615236 (560h39m15.066698271s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:08:15.195619865 (560h39m15.0667029s) [upload=491536]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:08:15.494969310 (560h39m15.366052355s) [upload=491536]: Done uploading segments...
Dec 29 00:08:15.494993957 (560h39m15.366076992s) [upload=491536]: Upload finished. err: <nil>
Duration: 1m0.003986361s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [X.X....X.....X.....X......X...X.............................]
duration: 
6.142626654s
trace: 
https://jaeger.dev.storj.io/trace/156ad5a012a53700
FAIL (2025-12-29 00:08:36 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.189688552 (560h19m2.060771587s) [upload=491232]: Starting upload
Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload
Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments...
Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false
Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment...
Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment.
Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment
Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments...
Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces.
Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection
Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled
Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments...
Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil>
Duration: 6.142654217s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [.X.X....X.....X.....X......X...X............................]
duration: 
6.027900111s
trace: 
https://jaeger.dev.storj.io/trace/156ad5a012a53700
FAIL (2025-12-29 00:07:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.189688552 (560h19m2.060771587s) [upload=491232]: Starting upload
Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload
Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments...
Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false
Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment...
Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment.
Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment
Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments...
Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces.
Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection
Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled
Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments...
Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil>
Duration: 6.027919147s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [........X...............................................X...]
duration: 
18.259755593s
trace: 
https://jaeger.dev.storj.io/trace/6bf1ab77f4378ccd
FAIL (2025-12-29 00:07:23 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
 [noise=true , address=145.79.227.40:4060   , piece_id=O3DYON2GBH2DLOGCBCVOAP4CTWXAKQSBAQ5VIFHTGQHTA7V6VSEA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.214957292 (560h35m27.086040337s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.215553046 (560h35m27.086636111s) [noise=true , address=145.79.227.21:4495   , piece_id=VAPESBYPOCAHSAKVUF4BX6S7C3V57UN52JMLG5RKZXXCMUN5GGUQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216046225 (560h35m27.08712927s) [noise=true , address=77.90.31.15:4727     , piece_id=I6SPG4DPO7XWSJ7BHKWGU5XKV3FKNG242TWKMMXBYUZ367BVWHRQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216522101 (560h35m27.087605146s) [noise=true , address=145.79.227.35:4495   , piece_id=RAYUA6ULAIEB5JFPSS5VNICEBJ7TQECP4BW54XYVWZ2DKJRYXPBA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216589970 (560h35m27.087673005s) [noise=true , address=77.90.31.18:4834     , piece_id=U6S5SHRGYUWKMGASJEHQV7CUBS2YSXDJNUJS3YY74MRAZFIZX2YQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216943142 (560h35m27.088026187s) [noise=true , address=145.79.227.18:4036   , piece_id=3DFTC3CKH72236XY2KXTGPCCWWIAG2IVVPA2ERJAUKPWCEH7JHBA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216966367 (560h35m27.088049402s) [noise=true , address=145.79.227.26:4676   , piece_id=YUZDAOQYBX2Q3RI422HNEL6Z3DM7CEFS73CCAQZQM3CWBO4FYPWA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217409611 (560h35m27.088492646s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217453494 (560h35m27.088536539s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217790846 (560h35m27.088873891s) [noise=true , address=77.90.31.13:4654     , piece_id=IWOSKEZW4KNSQA7BJEO6ZZHT4VOXWMVYIUTQMJ5BIFMNBLTXS72Q, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217854988 (560h35m27.088938023s) [noise=true , address=145.79.227.44:4212   , piece_id=V2JEEO23QWC3JQ3DJ22LMBTOAZD4WPZLFDR53FPMXNWMFV6QCG7Q, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218185508 (560h35m27.089268543s) [noise=true , address=45.91.116.14:4198    , piece_id=OJFOMJQCFL53RGOQKG6H5N5YY4EQKBWCYML4UNF36AHU5KEQ3RTQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218299645 (560h35m27.08938268s) [noise=true , address=45.91.116.12:4240    , piece_id=JKYZ4TGQ4H5YUKRFTF4APL6DLOCAHAU5V5647CHTXWK4YHTCHRPA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218513703 (560h35m27.089596748s) [noise=true , address=145.79.227.46:4634   , piece_id=K5ZE7O2NC6IL6BFYISQLCM2FEXHNNLV74UPK2T3Y4CAMUVKGANVA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218685610 (560h35m27.089768655s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:28.677241858 (560h35m28.548324893s) [upload=491486]: Next returned nil segment
Dec 29 00:04:28.677256546 (560h35m28.548339591s) [upload=491486]: Waiting for error group managing segments...
Dec 29 00:04:29.085059743 (560h35m28.956142788s) [noise=true , address=145.79.227.26:4676   , piece_id=YUZDAOQYBX2Q3RI422HNEL6Z3DM7CEFS73CCAQZQM3CWBO4FYPWA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.087003614 (560h35m28.958086649s) [noise=true , address=145.79.227.18:4036   , piece_id=3DFTC3CKH72236XY2KXTGPCCWWIAG2IVVPA2ERJAUKPWCEH7JHBA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.088949909 (560h35m28.960032944s) [noise=true , address=145.79.227.21:4495   , piece_id=VAPESBYPOCAHSAKVUF4BX6S7C3V57UN52JMLG5RKZXXCMUN5GGUQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.188360182 (560h35m29.059443217s) [noise=true , address=145.79.227.46:4634   , piece_id=K5ZE7O2NC6IL6BFYISQLCM2FEXHNNLV74UPK2T3Y4CAMUVKGANVA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.230588512 (560h35m29.101671547s) [noise=true , address=145.79.227.29:4960   , piece_id=YEW2LBDZXLYYKWM6SVVZJNBJ4SNUUIERSYDJFLXFKA66LELIXNAQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.242296684 (560h35m29.113379719s) [noise=true , address=77.90.31.13:4654     , piece_id=IWOSKEZW4KNSQA7BJEO6ZZHT4VOXWMVYIUTQMJ5BIFMNBLTXS72Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.259417151 (560h35m29.130500186s) [noise=true , address=77.90.31.18:4834     , piece_id=U6S5SHRGYUWKMGASJEHQV7CUBS2YSXDJNUJS3YY74MRAZFIZX2YQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.264333110 (560h35m29.135416155s) [noise=true , address=77.90.31.15:4727     , piece_id=I6SPG4DPO7XWSJ7BHKWGU5XKV3FKNG242TWKMMXBYUZ367BVWHRQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.304739492 (560h35m29.175822527s) [noise=true , address=77.90.31.10:4530     , piece_id=L7XPYQVLGOIVJTCIGHYGZJRL3P3NG5XFCUJYSLZPLGYHHIAMWF3Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.315129384 (560h35m29.186212429s) [noise=true , address=45.91.116.12:4240    , piece_id=JKYZ4TGQ4H5YUKRFTF4APL6DLOCAHAU5V5647CHTXWK4YHTCHRPA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.504670231 (560h35m29.375753276s) [noise=true , address=45.91.116.14:4198    , piece_id=OJFOMJQCFL53RGOQKG6H5N5YY4EQKBWCYML4UNF36AHU5KEQ3RTQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.972771571 (560h35m29.843854606s) [noise=true , address=145.79.227.40:4060   , piece_id=O3DYON2GBH2DLOGCBCVOAP4CTWXAKQSBAQ5VIFHTGQHTA7V6VSEA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.056334381 (560h35m29.927417416s) [noise=true , address=145.79.227.35:4495   , piece_id=RAYUA6ULAIEB5JFPSS5VNICEBJ7TQECP4BW54XYVWZ2DKJRYXPBA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.059972358 (560h35m29.931055393s) [noise=true , address=145.79.227.44:4212   , piece_id=V2JEEO23QWC3JQ3DJ22LMBTOAZD4WPZLFDR53FPMXNWMFV6QCG7Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.059987877 (560h35m29.931070912s) [seg_pos={0 0}, upload=491486]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:04:30.060151148 (560h35m29.931234183s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060158402 (560h35m29.931241437s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060264263 (560h35m29.931347298s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pDw6bqNwi4PSPjpnMHzAxZFoiFYTnGRUUb1a9u2D22csh6WmW): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060270305 (560h35m29.93135335s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060285264 (560h35m29.931368299s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection
Dec 29 00:04:30.060295833 (560h35m29.931378909s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060401775 (560h35m29.93148482s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060408818 (560h35m29.931491863s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060449657 (560h35m29.931532692s) [seg_pos={0 0}, upload=491486]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:04:30.060529128 (560h35m29.931612163s) [upload=491486]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:04:30.060535620 (560h35m29.931618655s) [upload=491486]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:04:30.363907475 (560h35m30.23499051s) [upload=491486]: Done uploading segments...
Dec 29 00:04:30.363925379 (560h35m30.235008424s) [upload=491486]: Upload finished. err: <nil>
Duration: 18.25978534s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
7 out of 60 is failed (threshold=3) [..X.X....X.....X.....X......X...X...........................]
duration: 
6.073704707s
trace: 
https://jaeger.dev.storj.io/trace/156ad5a012a53700
FAIL (2025-12-29 00:06:53 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.189688552 (560h19m2.060771587s) [upload=491232]: Starting upload
Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload
Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments...
Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false
Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment...
Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment.
Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment
Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments...
Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces.
Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection
Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled
Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments...
Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil>
Duration: 6.073725597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [X..X.X....X.....X.....X......X...X..........................]
duration: 
21.743596679s
trace: 
https://jaeger.dev.storj.io/trace/156ad5a012a53700
FAIL (2025-12-29 00:06:47 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.189688552 (560h19m2.060771587s) [upload=491232]: Starting upload
Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload
Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments...
Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false
Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment...
Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment.
Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment
Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments...
Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces.
Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection
Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled
Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments...
Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil>
Duration: 21.743625324s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
2 out of 60 is failed (threshold=3) [.........X...............................................X..]
duration: 
19.72064532s
trace: 
https://jaeger.dev.storj.io/trace/6bf1ab77f4378ccd
FAIL (2025-12-29 00:06:26 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
 [noise=true , address=145.79.227.40:4060   , piece_id=O3DYON2GBH2DLOGCBCVOAP4CTWXAKQSBAQ5VIFHTGQHTA7V6VSEA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.214957292 (560h35m27.086040337s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.215553046 (560h35m27.086636111s) [noise=true , address=145.79.227.21:4495   , piece_id=VAPESBYPOCAHSAKVUF4BX6S7C3V57UN52JMLG5RKZXXCMUN5GGUQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216046225 (560h35m27.08712927s) [noise=true , address=77.90.31.15:4727     , piece_id=I6SPG4DPO7XWSJ7BHKWGU5XKV3FKNG242TWKMMXBYUZ367BVWHRQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216522101 (560h35m27.087605146s) [noise=true , address=145.79.227.35:4495   , piece_id=RAYUA6ULAIEB5JFPSS5VNICEBJ7TQECP4BW54XYVWZ2DKJRYXPBA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216589970 (560h35m27.087673005s) [noise=true , address=77.90.31.18:4834     , piece_id=U6S5SHRGYUWKMGASJEHQV7CUBS2YSXDJNUJS3YY74MRAZFIZX2YQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216943142 (560h35m27.088026187s) [noise=true , address=145.79.227.18:4036   , piece_id=3DFTC3CKH72236XY2KXTGPCCWWIAG2IVVPA2ERJAUKPWCEH7JHBA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.216966367 (560h35m27.088049402s) [noise=true , address=145.79.227.26:4676   , piece_id=YUZDAOQYBX2Q3RI422HNEL6Z3DM7CEFS73CCAQZQM3CWBO4FYPWA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217409611 (560h35m27.088492646s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217453494 (560h35m27.088536539s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217790846 (560h35m27.088873891s) [noise=true , address=77.90.31.13:4654     , piece_id=IWOSKEZW4KNSQA7BJEO6ZZHT4VOXWMVYIUTQMJ5BIFMNBLTXS72Q, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.217854988 (560h35m27.088938023s) [noise=true , address=145.79.227.44:4212   , piece_id=V2JEEO23QWC3JQ3DJ22LMBTOAZD4WPZLFDR53FPMXNWMFV6QCG7Q, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218185508 (560h35m27.089268543s) [noise=true , address=45.91.116.14:4198    , piece_id=OJFOMJQCFL53RGOQKG6H5N5YY4EQKBWCYML4UNF36AHU5KEQ3RTQ, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218299645 (560h35m27.08938268s) [noise=true , address=45.91.116.12:4240    , piece_id=JKYZ4TGQ4H5YUKRFTF4APL6DLOCAHAU5V5647CHTXWK4YHTCHRPA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218513703 (560h35m27.089596748s) [noise=true , address=145.79.227.46:4634   , piece_id=K5ZE7O2NC6IL6BFYISQLCM2FEXHNNLV74UPK2T3Y4CAMUVKGANVA, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:27.218685610 (560h35m27.089768655s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Uploading piece...
Dec 29 00:04:28.677241858 (560h35m28.548324893s) [upload=491486]: Next returned nil segment
Dec 29 00:04:28.677256546 (560h35m28.548339591s) [upload=491486]: Waiting for error group managing segments...
Dec 29 00:04:29.085059743 (560h35m28.956142788s) [noise=true , address=145.79.227.26:4676   , piece_id=YUZDAOQYBX2Q3RI422HNEL6Z3DM7CEFS73CCAQZQM3CWBO4FYPWA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.087003614 (560h35m28.958086649s) [noise=true , address=145.79.227.18:4036   , piece_id=3DFTC3CKH72236XY2KXTGPCCWWIAG2IVVPA2ERJAUKPWCEH7JHBA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.088949909 (560h35m28.960032944s) [noise=true , address=145.79.227.21:4495   , piece_id=VAPESBYPOCAHSAKVUF4BX6S7C3V57UN52JMLG5RKZXXCMUN5GGUQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.188360182 (560h35m29.059443217s) [noise=true , address=145.79.227.46:4634   , piece_id=K5ZE7O2NC6IL6BFYISQLCM2FEXHNNLV74UPK2T3Y4CAMUVKGANVA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.230588512 (560h35m29.101671547s) [noise=true , address=145.79.227.29:4960   , piece_id=YEW2LBDZXLYYKWM6SVVZJNBJ4SNUUIERSYDJFLXFKA66LELIXNAQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.242296684 (560h35m29.113379719s) [noise=true , address=77.90.31.13:4654     , piece_id=IWOSKEZW4KNSQA7BJEO6ZZHT4VOXWMVYIUTQMJ5BIFMNBLTXS72Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.259417151 (560h35m29.130500186s) [noise=true , address=77.90.31.18:4834     , piece_id=U6S5SHRGYUWKMGASJEHQV7CUBS2YSXDJNUJS3YY74MRAZFIZX2YQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.264333110 (560h35m29.135416155s) [noise=true , address=77.90.31.15:4727     , piece_id=I6SPG4DPO7XWSJ7BHKWGU5XKV3FKNG242TWKMMXBYUZ367BVWHRQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.304739492 (560h35m29.175822527s) [noise=true , address=77.90.31.10:4530     , piece_id=L7XPYQVLGOIVJTCIGHYGZJRL3P3NG5XFCUJYSLZPLGYHHIAMWF3Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.315129384 (560h35m29.186212429s) [noise=true , address=45.91.116.12:4240    , piece_id=JKYZ4TGQ4H5YUKRFTF4APL6DLOCAHAU5V5647CHTXWK4YHTCHRPA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.504670231 (560h35m29.375753276s) [noise=true , address=45.91.116.14:4198    , piece_id=OJFOMJQCFL53RGOQKG6H5N5YY4EQKBWCYML4UNF36AHU5KEQ3RTQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:29.972771571 (560h35m29.843854606s) [noise=true , address=145.79.227.40:4060   , piece_id=O3DYON2GBH2DLOGCBCVOAP4CTWXAKQSBAQ5VIFHTGQHTA7V6VSEA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.056334381 (560h35m29.927417416s) [noise=true , address=145.79.227.35:4495   , piece_id=RAYUA6ULAIEB5JFPSS5VNICEBJ7TQECP4BW54XYVWZ2DKJRYXPBA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.059972358 (560h35m29.931055393s) [noise=true , address=145.79.227.44:4212   , piece_id=V2JEEO23QWC3JQ3DJ22LMBTOAZD4WPZLFDR53FPMXNWMFV6QCG7Q, seg_pos={0 0}, upload=491486]: Done uploading piece. err: <nil>
Dec 29 00:04:30.059987877 (560h35m29.931070912s) [seg_pos={0 0}, upload=491486]: Segment reached optimal threshold of 14 pieces.
Dec 29 00:04:30.060151148 (560h35m29.931234183s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060158402 (560h35m29.931241437s) [noise=true , address=145.79.227.35:4507   , piece_id=G5C4L5SNADYEL7UODPCNNZ4EWFY5SEMZPXA2L7JHEJEQEAUIXBLQ, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060264263 (560h35m29.931347298s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pDw6bqNwi4PSPjpnMHzAxZFoiFYTnGRUUb1a9u2D22csh6WmW): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060270305 (560h35m29.93135335s) [noise=true , address=45.91.116.7:4038     , piece_id=PFFNTMZE7BCOIQW4UQIHIHUSOPAMS5RJH5ICL63RHABDJLKI7L7A, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060285264 (560h35m29.931368299s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53748->64.49.13.169:4716: use of closed network connection
Dec 29 00:04:30.060295833 (560h35m29.931378909s) [noise=true , address=64.49.13.169:4716    , piece_id=EV4VDASXOSVNJKZDPWZSSU6LM75FSIZ2AX7BQM74H4LMHME6IBOA, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060401775 (560h35m29.93148482s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1g7r2boMdct1YvKxHZ5U3RM923N4UDfGNKGtYyAm6VKwfKAJn4): protocol: expected piece hash; context canceled; EOF
Dec 29 00:04:30.060408818 (560h35m29.931491863s) [noise=true , address=145.79.227.30:4768   , piece_id=UGW5YKMSNDAZICNYWR56POJD5VTUYVKZWKWY26PDK5EDCMRVYIGQ, seg_pos={0 0}, upload=491486]: Long tail cancellation due to optimal threshold
Dec 29 00:04:30.060449657 (560h35m29.931532692s) [seg_pos={0 0}, upload=491486]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 00:04:30.060529128 (560h35m29.931612163s) [upload=491486]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 00:04:30.060535620 (560h35m29.931618655s) [upload=491486]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 00:04:30.363907475 (560h35m30.23499051s) [upload=491486]: Done uploading segments...
Dec 29 00:04:30.363925379 (560h35m30.235008424s) [upload=491486]: Upload finished. err: <nil>
Duration: 19.720673512s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
8 out of 60 is failed (threshold=3) [.X..X.X....X.....X.....X......X...X.........................]
duration: 
6.018424465s
trace: 
https://jaeger.dev.storj.io/trace/156ad5a012a53700
FAIL (2025-12-29 00:06:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
.189688552 (560h19m2.060771587s) [upload=491232]: Starting upload
Dec 28 23:48:02.189755559 (560h19m2.060838635s) [upload=491232]: Done starting upload
Dec 28 23:48:02.190006468 (560h19m2.061089513s) [upload=491232]: Uploading segments...
Dec 28 23:48:02.190188124 (560h19m2.061271169s) [upload=491232]: Got next segment. Inline: false
Dec 28 23:48:02.190233109 (560h19m2.061316154s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 23:48:02.190273025 (560h19m2.06135607s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 23:48:02.436037429 (560h19m2.307120484s) [seg_pos={0 0}, upload=491232]: Begin upload segment...
Dec 28 23:48:02.436096522 (560h19m2.307179567s) [seg_pos={0 0}, upload=491232]: Done begin upload segment.
Dec 28 23:48:02.436134223 (560h19m2.307217269s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436225517 (560h19m2.307308562s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.436858271 (560h19m2.307941316s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437428267 (560h19m2.308511312s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.437873595 (560h19m2.30895663s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438299756 (560h19m2.309382801s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438646095 (560h19m2.30972914s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.438985572 (560h19m2.310068627s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.439374162 (560h19m2.310457197s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.441292164 (560h19m2.312375209s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.443086089 (560h19m2.314169134s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.444303947 (560h19m2.315386992s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.449230528 (560h19m2.320313563s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.453625756 (560h19m2.324708791s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.455381218 (560h19m2.326464263s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:02.458636065 (560h19m2.32971911s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Uploading piece...
Dec 28 23:48:04.140029585 (560h19m4.01111263s) [upload=491232]: Next returned nil segment
Dec 28 23:48:04.140059501 (560h19m4.011142556s) [upload=491232]: Waiting for error group managing segments...
Dec 28 23:48:04.442417125 (560h19m4.31350017s) [noise=true , address=145.79.227.37:4566   , piece_id=KTQIR37E3IEHNIJIPNOABBPNYLQYJN7N3GSM6TS5E6DBQ34YK5TQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.448605607 (560h19m4.319688663s) [noise=true , address=145.79.227.45:4230   , piece_id=QPNQHCZJNB5CJB6ECX2NCYVTYO5SINW3MOUD4UEUJ3QZGYKVB2CA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.454417563 (560h19m4.325500608s) [noise=true , address=145.79.227.48:4303   , piece_id=WXOJLAVIXD6SWCWOMOT3V3GLVYZNAR7J6VB4GUIIKGBP3MAWAN4A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.457909071 (560h19m4.328992106s) [noise=true , address=145.79.227.11:4730   , piece_id=34HZIFY3YOUDZYEZXOKT3VY6TQ6RTZ4VZBMVGURNHV5PGHVWBM5Q, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.463941576 (560h19m4.335024621s) [noise=true , address=145.79.227.39:4015   , piece_id=C7C22S6D5GHTB5ZESG7TY74CXMWGKMYJJA5Z2O5HEM4DCRKAMUPQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.464866538 (560h19m4.335949583s) [noise=true , address=145.79.227.38:4978   , piece_id=XLATLKGUDZ4TY6K7FRGO3IWFVU5R6IGC6SPAENKGXX54YQQ75FUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.505978793 (560h19m4.377061838s) [noise=true , address=145.79.227.16:4910   , piece_id=WVEUD5R4XMHGG5JVSYMPD74MYTUBM44VM5UKLSMHJRFIAAHCW7JQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.664354503 (560h19m4.535437548s) [noise=true , address=77.90.31.21:4877     , piece_id=SRSCLYXVOWXO2MP7BCFNNYG2AYDVLBCWIT6IOVONAGPIOR5WDYPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684213573 (560h19m4.555296619s) [noise=true , address=45.91.116.10:4083    , piece_id=THXUYIQWZ2AUNJARL2CLFJEZDZOINCEJ4CPREMURMMFNVC4PH7VA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.684766506 (560h19m4.555849542s) [noise=true , address=45.91.116.11:4119    , piece_id=L2R4I4NB6G3QLFLSMQKPGNUOHM2ZLGUU4NINWK7SN4QYYCUMTB7A, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.704586343 (560h19m4.575669388s) [noise=true , address=45.91.116.8:4046     , piece_id=G4KKCI4UALBP2Q2DSGLYDMV2QNV6QVVSX3VCZPDGKWPRC33R3HRQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:04.754571160 (560h19m4.625654195s) [noise=true , address=45.91.116.5:4298     , piece_id=VI2PKFUKHPTWMFV2YDTBKOHQXRIFP52WPXNMWPQJAITIS7QIPQOQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073649620 (560h19m4.944732665s) [noise=true , address=45.91.116.6:4974     , piece_id=7LXKAMBM22OY5R2PDO63DQ6MRGECFIIEV53FWEMF7ZWKI4OREQPA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: <nil>
Dec 28 23:48:05.073672603 (560h19m4.944755638s) [seg_pos={0 0}, upload=491232]: Segment reached optimal threshold of 13 pieces.
Dec 28 23:48:05.073959290 (560h19m4.945042325s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SqSapYVaJiXrqqGfD9a1XFgf6Y8eGJXmudUmLhX7yjmYXBWTm): protocol: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60230->145.79.227.41:4086: use of closed network connection
Dec 28 23:48:05.073967495 (560h19m4.94505054s) [noise=true , address=145.79.227.41:4086   , piece_id=FMO7IHFSLEHFK2ITRHN3Z32PSPUM3ESXPOPNTKPJ7CO74LN4WLEQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074031277 (560h19m4.945114322s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa): protocol: expected piece hash; context canceled; EOF
Dec 28 23:48:05.074047657 (560h19m4.945130692s) [noise=true , address=145.79.227.44:4195   , piece_id=IWOVA35AXW62FDF2V4L3CQGWY52HIIGIX5J4NCP25JWZVQ3OF5AA, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074235696 (560h19m4.945318741s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e): protocol: write tcp 188.166.225.170:40942->145.79.227.7:4294: use of closed network connection; context canceled
Dec 28 23:48:05.074243310 (560h19m4.945326355s) [noise=true , address=145.79.227.7:4294    , piece_id=OCF56WFEOGT5JUPDAUSQVOUE4UECHSFO3JOPJQOZTGHQ64KWJGUQ, seg_pos={0 0}, upload=491232]: Long tail cancellation due to optimal threshold
Dec 28 23:48:05.074260302 (560h19m4.945343337s) [seg_pos={0 0}, upload=491232]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 23:48:05.074354382 (560h19m4.945437417s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 23:48:05.074361265 (560h19m4.94544432s) [upload=491232]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 23:48:05.372728512 (560h19m5.243811547s) [upload=491232]: Done uploading segments...
Dec 28 23:48:05.372750323 (560h19m5.243833368s) [upload=491232]: Upload finished. err: <nil>
Duration: 6.018451907s core uplink us-select-1