Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.902341998s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:10:32 UTC) orbiter warning: context deadline exceeded Duration: 5.90236948s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....XX....X...X.....X.....XXX.X......................X..X.XX] duration: 30.002686777s trace: https://jaeger.dev.storj.io/trace/694c34495d4a1a9c FAIL (2025-12-26 08:09:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 08:09:27.995905500 (496h40m27.866988545s) [upload=435932]: Starting upload Dec 26 08:09:27.995957480 (496h40m27.867040515s) [upload=435932]: Done starting upload Dec 26 08:09:27.996076125 (496h40m27.86715917s) [upload=435932]: Uploading segments... Dec 26 08:09:27.996164113 (496h40m27.867247158s) [upload=435932]: Got next segment. Inline: false Dec 26 08:09:27.996195343 (496h40m27.867278378s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:09:27.996200783 (496h40m27.867283818s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:09:28.245767360 (496h40m28.116850395s) [seg_pos={0 0}, upload=435932]: Begin upload segment... Dec 26 08:09:28.245852092 (496h40m28.116935127s) [seg_pos={0 0}, upload=435932]: Done begin upload segment. Dec 26 08:09:28.245893761 (496h40m28.116976796s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.245899321 (496h40m28.116982356s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.246772374 (496h40m28.117855409s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247111419 (496h40m28.118194454s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247467898 (496h40m28.118550943s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.247903156 (496h40m28.118986191s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248341361 (496h40m28.119424396s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.248762924 (496h40m28.119845959s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.249157806 (496h40m28.120240841s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250531121 (496h40m28.121614166s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.250948365 (496h40m28.1220314s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.251809304 (496h40m28.122892339s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253211865 (496h40m28.124294911s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.253946885 (496h40m28.12502992s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.258262212 (496h40m28.129345257s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:28.269630158 (496h40m28.140713193s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Uploading piece... Dec 26 08:09:29.978129477 (496h40m29.849212522s) [upload=435932]: Next returned nil segment Dec 26 08:09:29.978145057 (496h40m29.849228102s) [upload=435932]: Waiting for error group managing segments... Dec 26 08:09:30.266733918 (496h40m30.137816953s) [noise=true , address=145.79.227.41:4085 , piece_id=TUYVCK7YILQ3EXZSOQNOM3CBK3B7MSVUQU7377GAW6QS7VPXEPKA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.267100977 (496h40m30.138184023s) [noise=true , address=145.79.227.16:4911 , piece_id=6OROM3ANMBCJSOLREE7DH52JTXINIGTWPKQTYPMUWTQTDD6EW5DA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.273766499 (496h40m30.144849544s) [noise=true , address=145.79.227.46:4603 , piece_id=CVDBVWOQ4SY5QOXWEPCPGQM5CNFDTZVD67DBUSSEBKGXFPDKDNMQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.279601881 (496h40m30.150684926s) [noise=true , address=145.79.227.51:4410 , piece_id=J6FQSXKS5DWDLB5OQMYSVCKLEQOMOL73HVTXXXTL4ARBZQILVCPQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.285658013 (496h40m30.156741058s) [noise=true , address=145.79.227.19:4078 , piece_id=C5WY2NLSKRM3IYZWISLCNDUW6Y33L7IH6DVVX4ZZP2GIGO7MRNMA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.290868184 (496h40m30.161951239s) [noise=true , address=145.79.227.3:4149 , piece_id=EC3QBAHCXPQ7SSSQYBVM3S5KW4SSRGZUF3WHWMSR2U36DSBLWITA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.502516858 (496h40m30.373599903s) [noise=true , address=45.91.116.11:4118 , piece_id=27IMZC6QT6YAOPH6FZW5YISPRPWILTNFTPEEEFSRXC6ETBL6HNNA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.553673489 (496h40m30.424756524s) [noise=true , address=45.91.116.8:4047 , piece_id=BXCX32GYWGBWTEHO6RFD7OW7GVV44JXIIBHGCOU6J242WNFSEMSQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.606375212 (496h40m30.477458247s) [noise=true , address=45.91.116.9:4261 , piece_id=XIY37R3GGNYLVBFWMCKZFBCA732IMJS4OJXET74Y72IQMYOGGUUA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.608045683 (496h40m30.479128718s) [noise=true , address=45.91.116.14:4190 , piece_id=WAWW2IC7YVD3YT3NLAVM432M2QFOWG3KF6ULPRM2Q4V7EKVYYWSA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.613178417 (496h40m30.484261462s) [noise=true , address=45.91.116.5:4297 , piece_id=DY6U7X577KRNKA6GQWY3OCXYDQFY2ZBDWH7V3K5LMIOQ5TSWINEQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.619262832 (496h40m30.490345867s) [noise=true , address=45.91.116.12:4228 , piece_id=NUJ7H6RDKL53Q3BOCDORFE2HJPFQC6AYZXGMIS5FKE3QJ7ODM5QA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644609176 (496h40m30.515692211s) [noise=true , address=77.90.31.14:4664 , piece_id=L62VLUSGXFQOH7V5EXJH7TD67QFMAVDK632UIY7ATLSJS53X35FA, seg_pos={0 0}, upload=435932]: Done uploading piece. err: <nil> Dec 26 08:09:30.644627411 (496h40m30.515710456s) [seg_pos={0 0}, upload=435932]: Segment reached optimal threshold of 13 pieces. Dec 26 08:09:30.644812703 (496h40m30.515895738s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644819887 (496h40m30.515902922s) [noise=true , address=77.90.31.16:4735 , piece_id=76UCJITYHP37KOT2DHN7O6CZAMGW6Y3ZMRXAFKVQA2UUPWR5KP6A, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.644944875 (496h40m30.51602791s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.644963070 (496h40m30.516046115s) [noise=true , address=45.91.116.7:4009 , piece_id=HFTOXAKFC7NJWQ7AKOSBSX5IWGBIBJGT6VB3LPRWIWLE6AIS46LQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645036961 (496h40m30.516119996s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:30.645045367 (496h40m30.516128402s) [noise=true , address=45.91.116.10:4081 , piece_id=42VXGPLL2HIM4AZSIJPFNEM2OU22NBWQH6QG7CHETWRT3EJGC2ZQ, seg_pos={0 0}, upload=435932]: Long tail cancellation due to optimal threshold Dec 26 08:09:30.645093338 (496h40m30.516176373s) [seg_pos={0 0}, upload=435932]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:09:30.645284733 (496h40m30.516367768s) [upload=435932]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:09:30.645294180 (496h40m30.516377225s) [upload=435932]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:09:30.938761932 (496h40m30.809844967s) [upload=435932]: Done uploading segments... Dec 26 08:09:30.938782241 (496h40m30.809865276s) [upload=435932]: Upload finished. err: <nil> Duration: 30.002695293s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.210570034s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:09:44 UTC) orbiter warning: context deadline exceeded Duration: 6.210658493s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 3.71583842s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:09:39 UTC) orbiter warning: context deadline exceeded Duration: 3.715866122s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001188976s trace: https://jaeger.dev.storj.io/trace/4b8b74dd4906ed4 FAIL (2025-12-26 08:09:35 UTC) orbiter warning: context deadline exceeded Duration: 30.001213212s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....XX....X...X.....X.....XXX.X......................X..X.X] duration: 30.003382302s trace: https://jaeger.dev.storj.io/trace/2cc3cf45f0b4f40a FAIL (2025-12-26 08:09:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ) context deadline exceeded ======LOGS====== Dec 26 08:08:57.775505721 (496h39m57.646588756s) [upload=435927]: Starting upload Dec 26 08:08:57.775578029 (496h39m57.646661064s) [upload=435927]: Done starting upload Dec 26 08:08:57.775821673 (496h39m57.646904718s) [upload=435927]: Uploading segments... Dec 26 08:08:57.775975425 (496h39m57.64705847s) [upload=435927]: Got next segment. Inline: false Dec 26 08:08:57.776761893 (496h39m57.647844938s) [upload=435927]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:08:57.776791500 (496h39m57.647874545s) [upload=435927]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:08:58.050026344 (496h39m57.921109389s) [seg_pos={0 0}, upload=435927]: Begin upload segment... Dec 26 08:08:58.050092410 (496h39m57.921175455s) [seg_pos={0 0}, upload=435927]: Done begin upload segment. Dec 26 08:08:58.050162283 (496h39m57.921245318s) [noise=true , address=145.79.227.42:4123 , piece_id=IIAMXKL2WYPNZB2PHZEMVCLC7XIVTNJSPAEBMJCGLV2NUUWPIWBQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.050174325 (496h39m57.92125737s) [noise=true , address=45.91.116.14:4192 , piece_id=V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.051811232 (496h39m57.922894267s) [noise=true , address=145.79.227.38:4979 , piece_id=ISF2OUO6QI5GUW7SAHNIG7SHHCMELHIIXZIS24HJAM6QR3J4NG4Q, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.051953854 (496h39m57.923036899s) [noise=true , address=77.90.31.10:4519 , piece_id=FBXQL7VBVLIY4S7DVMG4A26PW6SI6NDLBWGBJOVZXOVKZHHNOJGQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052331854 (496h39m57.923414889s) [noise=true , address=45.91.116.8:4046 , piece_id=CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052438937 (496h39m57.923521982s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052726735 (496h39m57.92380977s) [noise=true , address=145.79.227.48:4301 , piece_id=6FRHKBQQGUOIRP6VAYWNKZKRT7BJJUQCKHOHSZBFNXFQNZM3346Q, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.052933098 (496h39m57.924016143s) [noise=true , address=145.79.227.45:4230 , piece_id=T2TST2XNNHPXOKQPFVX6UYTMJX7CB2GHSWKZ6UKKQDTY5FBN7GTA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.053113142 (496h39m57.924196187s) [noise=true , address=45.91.116.10:4084 , piece_id=ANRH3APVGCFKORPVBPBD22RHDYBYUZLIMNBAXMIUF7SHJ2PVNBFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.053419104 (496h39m57.924502139s) [noise=true , address=77.90.31.16:4736 , piece_id=AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.054406274 (496h39m57.925489309s) [noise=true , address=145.79.227.19:4078 , piece_id=QMU5C6DJATPUTHIRH47W3KCXFOWGSBVWY2G7VV45ALZPQEJKYZSQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.054990486 (496h39m57.926073521s) [noise=true , address=145.79.227.32:4835 , piece_id=YSQA4TXGWOOSRIIMSHBRP25JOAXGLMYOTW3JHACKKCSGICTHBXOA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.055052985 (496h39m57.92613602s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.055610176 (496h39m57.926693231s) [noise=true , address=145.79.227.5:4221 , piece_id=5L3INSDWHHGZHTWJ6IQVQKHXGP47PICZGUGH23V43PGX2H6JFHDA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.056751559 (496h39m57.927834605s) [noise=true , address=145.79.227.18:4007 , piece_id=LFXWJ3TTUVC5KZVJV4BUNJ2ML2IKCURJMMFNCY3ZR6PKQIPXDGPA, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:58.057311806 (496h39m57.928394841s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Uploading piece... Dec 26 08:08:59.765097205 (496h39m59.63618024s) [upload=435927]: Next returned nil segment Dec 26 08:08:59.765116653 (496h39m59.636199688s) [upload=435927]: Waiting for error group managing segments... Dec 26 08:09:00.050879524 (496h39m59.921962559s) [noise=true , address=145.79.227.18:4007 , piece_id=LFXWJ3TTUVC5KZVJV4BUNJ2ML2IKCURJMMFNCY3ZR6PKQIPXDGPA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.054702302 (496h39m59.925785337s) [noise=true , address=145.79.227.19:4078 , piece_id=QMU5C6DJATPUTHIRH47W3KCXFOWGSBVWY2G7VV45ALZPQEJKYZSQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.057171624 (496h39m59.928254669s) [noise=true , address=145.79.227.32:4835 , piece_id=YSQA4TXGWOOSRIIMSHBRP25JOAXGLMYOTW3JHACKKCSGICTHBXOA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.073574328 (496h39m59.944657363s) [noise=true , address=145.79.227.48:4301 , piece_id=6FRHKBQQGUOIRP6VAYWNKZKRT7BJJUQCKHOHSZBFNXFQNZM3346Q, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.074979714 (496h39m59.946062759s) [noise=true , address=145.79.227.38:4979 , piece_id=ISF2OUO6QI5GUW7SAHNIG7SHHCMELHIIXZIS24HJAM6QR3J4NG4Q, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.083596281 (496h39m59.954679316s) [noise=true , address=145.79.227.5:4221 , piece_id=5L3INSDWHHGZHTWJ6IQVQKHXGP47PICZGUGH23V43PGX2H6JFHDA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.084605803 (496h39m59.955688848s) [noise=true , address=145.79.227.42:4123 , piece_id=IIAMXKL2WYPNZB2PHZEMVCLC7XIVTNJSPAEBMJCGLV2NUUWPIWBQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.090001598 (496h39m59.961084653s) [noise=true , address=145.79.227.45:4230 , piece_id=T2TST2XNNHPXOKQPFVX6UYTMJX7CB2GHSWKZ6UKKQDTY5FBN7GTA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.306896503 (496h40m0.177979568s) [noise=true , address=45.91.116.14:4192 , piece_id=V7Y6RCNZ4LJAA2V76KDEFYFX3DTZYNM4B6LRZDZ4CAWCFDHHLCFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.320347926 (496h40m0.191430961s) [noise=true , address=45.91.116.8:4046 , piece_id=CXFZLHRRVMWK45GFJM7PDG7G4GWF5XOIBCIDHEAQWL2VX2WPIUDA, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.369413926 (496h40m0.240496961s) [noise=true , address=77.90.31.16:4736 , piece_id=AKOSKSLPETP6L24NFDADZJJZYJBVJ5CA2JRJI4XTE6RPXGQ2VHBQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.411558997 (496h40m0.282642042s) [noise=true , address=45.91.116.10:4084 , piece_id=ANRH3APVGCFKORPVBPBD22RHDYBYUZLIMNBAXMIUF7SHJ2PVNBFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.929994685 (496h40m0.801077731s) [noise=true , address=77.90.31.10:4519 , piece_id=FBXQL7VBVLIY4S7DVMG4A26PW6SI6NDLBWGBJOVZXOVKZHHNOJGQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: <nil> Dec 26 08:09:00.930018842 (496h40m0.801101887s) [seg_pos={0 0}, upload=435927]: Segment reached optimal threshold of 13 pieces. Dec 26 08:09:00.930554020 (496h40m0.801637066s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:00.930566916 (496h40m0.801649951s) [noise=true , address=77.90.31.21:4880 , piece_id=6AUYDALDN32IVYYSE2D72HGULS6TYY5YRZ3ISAYOZ4JMARIQUQFQ, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.930878820 (496h40m0.801961865s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF Dec 26 08:09:00.930890712 (496h40m0.801973757s) [noise=true , address=45.91.116.9:4263 , piece_id=MWFHF3TEJGENCED5Z5LMSJE5DFPOJSRLC6MWVOZZS6QYEUWZ7HOQ, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.930966696 (496h40m0.802049741s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: read tcp 188.166.225.170:34724->77.90.31.19:4844: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 26 08:09:00.930992526 (496h40m0.802075571s) [noise=true , address=77.90.31.19:4844 , piece_id=RNH7QMZTAGKJS7OQJ6KEOKM2AFAULB2T24UFPOGO5VDU3GEH5K7A, seg_pos={0 0}, upload=435927]: Long tail cancellation due to optimal threshold Dec 26 08:09:00.931018926 (496h40m0.802101961s) [seg_pos={0 0}, upload=435927]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:09:00.931162299 (496h40m0.802245344s) [upload=435927]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:09:00.931170925 (496h40m0.80225397s) [upload=435927]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:09:01.226100892 (496h40m1.097183947s) [upload=435927]: Done uploading segments... Dec 26 08:09:01.226158521 (496h40m1.097241566s) [upload=435927]: Upload finished. err: <nil> Duration: 30.003403192s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001419489s trace: https://jaeger.dev.storj.io/trace/185fddc4dc40eba9 FAIL (2025-12-26 08:09:20 UTC) orbiter warning: context deadline exceeded Duration: 5.001442734s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X.....XX....X...X.....X.....XXX.X......................X..X.] duration: 6.115222077s trace: https://jaeger.dev.storj.io/trace/30af03556a5513c5 FAIL (2025-12-26 08:07:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ) context deadline exceeded ======LOGS====== Dec 26 08:07:07.906767540 (496h38m7.777850585s) [upload=435902]: Starting upload Dec 26 08:07:07.906847712 (496h38m7.777930757s) [upload=435902]: Done starting upload Dec 26 08:07:07.906950288 (496h38m7.778033333s) [upload=435902]: Uploading segments... Dec 26 08:07:07.907089693 (496h38m7.778172738s) [upload=435902]: Got next segment. Inline: false Dec 26 08:07:07.907123157 (496h38m7.778206202s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:07:07.907128818 (496h38m7.778211863s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:07:08.156507276 (496h38m8.027590321s) [seg_pos={0 0}, upload=435902]: Begin upload segment... Dec 26 08:07:08.156552783 (496h38m8.027635818s) [seg_pos={0 0}, upload=435902]: Done begin upload segment. Dec 26 08:07:08.156588841 (496h38m8.027671886s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.156650349 (496h38m8.027733394s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157120024 (496h38m8.028203069s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157494075 (496h38m8.02857711s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157949754 (496h38m8.029032789s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.158274692 (496h38m8.029357737s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.160792296 (496h38m8.031875331s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.162033850 (496h38m8.033116895s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163141009 (496h38m8.034224054s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163484513 (496h38m8.034567558s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.164352194 (496h38m8.035435239s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165066104 (496h38m8.036149139s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165484070 (496h38m8.036567105s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.167330015 (496h38m8.03841306s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.168621784 (496h38m8.03970482s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.169725707 (496h38m8.040808742s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:09.349050539 (496h38m9.220133594s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.622296024 (496h38m9.493379079s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.893726172 (496h38m9.764809207s) [upload=435902]: Next returned nil segment Dec 26 08:07:09.893739698 (496h38m9.764822743s) [upload=435902]: Waiting for error group managing segments... Dec 26 08:07:10.176986808 (496h38m10.048069853s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.184413820 (496h38m10.055496865s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.189827909 (496h38m10.060910944s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.190219355 (496h38m10.0613024s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.202474541 (496h38m10.073557576s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.203448456 (496h38m10.074531491s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.206554119 (496h38m10.077637154s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.210487950 (496h38m10.081570985s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.229236820 (496h38m10.100319855s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.416501755 (496h38m10.2875848s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.724923190 (496h38m10.596006235s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.179810177 (496h38m12.050893222s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.480684464 (496h38m12.351767499s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:13.352037555 (496h38m13.2231206s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:13.352077571 (496h38m13.223160616s) [seg_pos={0 0}, upload=435902]: Segment reached optimal threshold of 13 pieces. Dec 26 08:07:13.352273996 (496h38m13.223357041s) [seg_pos={0 0}, upload=435902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:07:13.352363626 (496h38m13.223446671s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:07:13.352370519 (496h38m13.223453564s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:07:13.646491094 (496h38m13.517574129s) [upload=435902]: Done uploading segments... Dec 26 08:07:13.646512355 (496h38m13.51759539s) [upload=435902]: Upload finished. err: <nil> Duration: 6.115239611s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X.....XX....X...X.....X.....XXX.X......................X..X] duration: 30.004076735s trace: https://jaeger.dev.storj.io/trace/30af03556a5513c5 FAIL (2025-12-26 08:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ) context deadline exceeded ======LOGS====== Dec 26 08:07:07.906767540 (496h38m7.777850585s) [upload=435902]: Starting upload Dec 26 08:07:07.906847712 (496h38m7.777930757s) [upload=435902]: Done starting upload Dec 26 08:07:07.906950288 (496h38m7.778033333s) [upload=435902]: Uploading segments... Dec 26 08:07:07.907089693 (496h38m7.778172738s) [upload=435902]: Got next segment. Inline: false Dec 26 08:07:07.907123157 (496h38m7.778206202s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:07:07.907128818 (496h38m7.778211863s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:07:08.156507276 (496h38m8.027590321s) [seg_pos={0 0}, upload=435902]: Begin upload segment... Dec 26 08:07:08.156552783 (496h38m8.027635818s) [seg_pos={0 0}, upload=435902]: Done begin upload segment. Dec 26 08:07:08.156588841 (496h38m8.027671886s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.156650349 (496h38m8.027733394s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157120024 (496h38m8.028203069s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157494075 (496h38m8.02857711s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.157949754 (496h38m8.029032789s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.158274692 (496h38m8.029357737s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.160792296 (496h38m8.031875331s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.162033850 (496h38m8.033116895s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163141009 (496h38m8.034224054s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.163484513 (496h38m8.034567558s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.164352194 (496h38m8.035435239s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165066104 (496h38m8.036149139s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.165484070 (496h38m8.036567105s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.167330015 (496h38m8.03841306s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.168621784 (496h38m8.03970482s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:08.169725707 (496h38m8.040808742s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Uploading piece... Dec 26 08:07:09.349050539 (496h38m9.220133594s) [noise=true , address=77.90.31.12:4589 , piece_id=N7CZHZEIBLAGZKKXKPXERI2QIKIR424ZHJMJKVA36UF3SNLBL4FQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, address:77.90.31.12:4589): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.622296024 (496h38m9.493379079s) [noise=true , address=77.90.31.18:4805 , piece_id=KWP3F453PXGCWYGQBGJCMW4IIMYUUGWRNHYJRLN2VNWGDPQHCV2A, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:09.893726172 (496h38m9.764809207s) [upload=435902]: Next returned nil segment Dec 26 08:07:09.893739698 (496h38m9.764822743s) [upload=435902]: Waiting for error group managing segments... Dec 26 08:07:10.176986808 (496h38m10.048069853s) [noise=true , address=145.79.227.51:4409 , piece_id=4HRMOJJKF3K3ONPWY6YC7V3LOZ2MJMJBK2G36HULCMHJEUXLH2AQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.184413820 (496h38m10.055496865s) [noise=true , address=145.79.227.3:4150 , piece_id=OPEEED2VTFN3DGOMOC6PRAV3VC6XQTVLYGML7CXK26XQ5KAEV5MA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.189827909 (496h38m10.060910944s) [noise=true , address=145.79.227.19:4078 , piece_id=TLFK7MBN5OJERNXCXFDSVUIQUFF4A6654U52UK2RNS4Z2D7K2YLQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.190219355 (496h38m10.0613024s) [noise=true , address=145.79.227.17:4970 , piece_id=P3QDN2Z2LPS66PVL55JF4PEYHI3WYHNGVOFLV5EUUVDEP6TROURA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.202474541 (496h38m10.073557576s) [noise=true , address=145.79.227.11:4730 , piece_id=UX4BQ7EVZXIDMVX7PTUH67W3TXHJO52RR2KUF7VSRL2KMYAPVHSQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.203448456 (496h38m10.074531491s) [noise=true , address=145.79.227.4:4185 , piece_id=FBPU4HO6MRVSXGZZ6RU5N76Z7CUEZGGHW22FSDMZYWDGVHTGFSMQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.206554119 (496h38m10.077637154s) [noise=true , address=145.79.227.46:4603 , piece_id=JSL3WMUJCBNM6U3F3YXC23O42NEYWUYT6I5LFOFT6LCP6RZ6IBMA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.210487950 (496h38m10.081570985s) [noise=true , address=145.79.227.49:4339 , piece_id=IOYNBKYW3DHULYJDVMNH34F4GY7GRU7BOKO2FO4HPDLGESPHIYHA, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.229236820 (496h38m10.100319855s) [noise=true , address=145.79.227.47:4267 , piece_id=V37QLFJTKR72T7BUELFPHQJOIDK4NN44NPDVH3IKXDPLBQPR3VKQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.416501755 (496h38m10.2875848s) [noise=true , address=45.91.116.13:4155 , piece_id=UDRMOUCQDALXRIEYHGQXU5G7QZPFSDLCFIVITD7WMUBFOXZAI6KQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:10.724923190 (496h38m10.596006235s) [noise=true , address=45.91.116.12:4227 , piece_id=QUO6VRMLGILNAX4P2VZ7LF5ZJDQ2SUJZLVSXUZKN5RMJKCHUXXQQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.179810177 (496h38m12.050893222s) [noise=true , address=45.91.116.10:4084 , piece_id=S4XMDBJ2FGIKYNGJ6WMQEKHHBY4RNVVUJHVOPGC4VSFF5XBMB36Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:12.480684464 (496h38m12.351767499s) [noise=true , address=77.90.31.15:4697 , piece_id=AENGNHGGOJFQHE6KMQEZYMC6BEZCRNEGKOZG2G7VA3KYUSG3SBCQ, seg_pos={0 0}, upload=435902]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: storage node overloaded, request limit: 2000 Dec 26 08:07:13.352037555 (496h38m13.2231206s) [noise=true , address=77.90.31.16:4736 , piece_id=CHQHMCKJ6RLQO43OWJNIV2AFDZXDULKPBODP3IB7NCHQU2O5R45Q, seg_pos={0 0}, upload=435902]: Done uploading piece. err: <nil> Dec 26 08:07:13.352077571 (496h38m13.223160616s) [seg_pos={0 0}, upload=435902]: Segment reached optimal threshold of 13 pieces. Dec 26 08:07:13.352273996 (496h38m13.223357041s) [seg_pos={0 0}, upload=435902]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:07:13.352363626 (496h38m13.223446671s) [upload=435902]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:07:13.352370519 (496h38m13.223453564s) [upload=435902]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:07:13.646491094 (496h38m13.517574129s) [upload=435902]: Done uploading segments... Dec 26 08:07:13.646512355 (496h38m13.51759539s) [upload=435902]: Upload finished. err: <nil> Duration: 30.004106191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X.....XX....X...X.....X.....XXX.X......................X..] duration: 6.074043776s trace: https://jaeger.dev.storj.io/trace/4718c0a3d5f7d0b6 FAIL (2025-12-26 08:06:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : Starting upload Dec 26 08:05:59.308561696 (496h36m59.179644761s) [upload=435884]: Done starting upload Dec 26 08:05:59.308762437 (496h36m59.179845482s) [upload=435884]: Uploading segments... Dec 26 08:05:59.308803346 (496h36m59.179886401s) [upload=435884]: Got next segment. Inline: false Dec 26 08:05:59.308828523 (496h36m59.179911568s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:05:59.308832941 (496h36m59.179915976s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:05:59.554822597 (496h36m59.425905752s) [seg_pos={0 0}, upload=435884]: Begin upload segment... Dec 26 08:05:59.554992110 (496h36m59.426075155s) [seg_pos={0 0}, upload=435884]: Done begin upload segment. Dec 26 08:05:59.555001227 (496h36m59.426084272s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555272563 (496h36m59.426355609s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555831007 (496h36m59.426914052s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.557563516 (496h36m59.428646551s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558176474 (496h36m59.429259529s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558605560 (496h36m59.429688605s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558918105 (496h36m59.430001141s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560223281 (496h36m59.431306326s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560497062 (496h36m59.431580097s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.561239075 (496h36m59.43232211s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562107749 (496h36m59.433190795s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562234630 (496h36m59.433317676s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562980201 (496h36m59.434063246s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564012245 (496h36m59.43509529s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564629551 (496h36m59.435712596s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.565120215 (496h36m59.43620325s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:06:01.176850499 (496h37m1.047933554s) [upload=435884]: Next returned nil segment Dec 26 08:06:01.176869015 (496h37m1.04795206s) [upload=435884]: Waiting for error group managing segments... Dec 26 08:06:01.560358332 (496h37m1.431441387s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.561209874 (496h37m1.432292919s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.567872280 (496h37m1.438955325s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.568944903 (496h37m1.440027948s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.570504894 (496h37m1.441587939s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588111790 (496h37m1.459194836s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588807835 (496h37m1.45989088s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.591812196 (496h37m1.462895241s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.594307557 (496h37m1.465390602s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.597042546 (496h37m1.468125591s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.763606471 (496h37m1.634689516s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.911658847 (496h37m1.782741892s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032411771 (496h37m1.903494826s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032436208 (496h37m1.903519253s) [seg_pos={0 0}, upload=435884]: Segment reached optimal threshold of 13 pieces. Dec 26 08:06:02.032691944 (496h37m1.90377499s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection Dec 26 08:06:02.032701914 (496h37m1.903784959s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032730317 (496h37m1.903813373s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 08:06:02.032740047 (496h37m1.903823102s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032853382 (496h37m1.903936427s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44234->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 08:06:02.032861668 (496h37m1.903944723s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032948333 (496h37m1.904031388s) [seg_pos={0 0}, upload=435884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:06:02.033105061 (496h37m1.904188116s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:06:02.033140599 (496h37m1.904223654s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:06:02.328202557 (496h37m2.199285602s) [upload=435884]: Done uploading segments... Dec 26 08:06:02.328232714 (496h37m2.199315759s) [upload=435884]: Upload finished. err: <nil> Duration: 6.07407761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X..X.....XX....X...X.....X.....XXX.X......................X.] duration: 6.046440023s trace: https://jaeger.dev.storj.io/trace/4718c0a3d5f7d0b6 FAIL (2025-12-26 08:06:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : Starting upload Dec 26 08:05:59.308561696 (496h36m59.179644761s) [upload=435884]: Done starting upload Dec 26 08:05:59.308762437 (496h36m59.179845482s) [upload=435884]: Uploading segments... Dec 26 08:05:59.308803346 (496h36m59.179886401s) [upload=435884]: Got next segment. Inline: false Dec 26 08:05:59.308828523 (496h36m59.179911568s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:05:59.308832941 (496h36m59.179915976s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:05:59.554822597 (496h36m59.425905752s) [seg_pos={0 0}, upload=435884]: Begin upload segment... Dec 26 08:05:59.554992110 (496h36m59.426075155s) [seg_pos={0 0}, upload=435884]: Done begin upload segment. Dec 26 08:05:59.555001227 (496h36m59.426084272s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555272563 (496h36m59.426355609s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555831007 (496h36m59.426914052s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.557563516 (496h36m59.428646551s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558176474 (496h36m59.429259529s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558605560 (496h36m59.429688605s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558918105 (496h36m59.430001141s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560223281 (496h36m59.431306326s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560497062 (496h36m59.431580097s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.561239075 (496h36m59.43232211s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562107749 (496h36m59.433190795s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562234630 (496h36m59.433317676s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562980201 (496h36m59.434063246s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564012245 (496h36m59.43509529s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564629551 (496h36m59.435712596s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.565120215 (496h36m59.43620325s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:06:01.176850499 (496h37m1.047933554s) [upload=435884]: Next returned nil segment Dec 26 08:06:01.176869015 (496h37m1.04795206s) [upload=435884]: Waiting for error group managing segments... Dec 26 08:06:01.560358332 (496h37m1.431441387s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.561209874 (496h37m1.432292919s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.567872280 (496h37m1.438955325s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.568944903 (496h37m1.440027948s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.570504894 (496h37m1.441587939s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588111790 (496h37m1.459194836s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588807835 (496h37m1.45989088s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.591812196 (496h37m1.462895241s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.594307557 (496h37m1.465390602s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.597042546 (496h37m1.468125591s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.763606471 (496h37m1.634689516s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.911658847 (496h37m1.782741892s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032411771 (496h37m1.903494826s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032436208 (496h37m1.903519253s) [seg_pos={0 0}, upload=435884]: Segment reached optimal threshold of 13 pieces. Dec 26 08:06:02.032691944 (496h37m1.90377499s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection Dec 26 08:06:02.032701914 (496h37m1.903784959s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032730317 (496h37m1.903813373s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 08:06:02.032740047 (496h37m1.903823102s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032853382 (496h37m1.903936427s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44234->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 08:06:02.032861668 (496h37m1.903944723s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032948333 (496h37m1.904031388s) [seg_pos={0 0}, upload=435884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:06:02.033105061 (496h37m1.904188116s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:06:02.033140599 (496h37m1.904223654s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:06:02.328202557 (496h37m2.199285602s) [upload=435884]: Done uploading segments... Dec 26 08:06:02.328232714 (496h37m2.199315759s) [upload=435884]: Upload finished. err: <nil> Duration: 6.046457345s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X..X.....XX....X...X.....X.....XXX.X......................X] duration: 30.00231391s trace: https://jaeger.dev.storj.io/trace/4718c0a3d5f7d0b6 FAIL (2025-12-26 08:06:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== : Starting upload Dec 26 08:05:59.308561696 (496h36m59.179644761s) [upload=435884]: Done starting upload Dec 26 08:05:59.308762437 (496h36m59.179845482s) [upload=435884]: Uploading segments... Dec 26 08:05:59.308803346 (496h36m59.179886401s) [upload=435884]: Got next segment. Inline: false Dec 26 08:05:59.308828523 (496h36m59.179911568s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 08:05:59.308832941 (496h36m59.179915976s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 08:05:59.554822597 (496h36m59.425905752s) [seg_pos={0 0}, upload=435884]: Begin upload segment... Dec 26 08:05:59.554992110 (496h36m59.426075155s) [seg_pos={0 0}, upload=435884]: Done begin upload segment. Dec 26 08:05:59.555001227 (496h36m59.426084272s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555272563 (496h36m59.426355609s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.555831007 (496h36m59.426914052s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.557563516 (496h36m59.428646551s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558176474 (496h36m59.429259529s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558605560 (496h36m59.429688605s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.558918105 (496h36m59.430001141s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560223281 (496h36m59.431306326s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.560497062 (496h36m59.431580097s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.561239075 (496h36m59.43232211s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562107749 (496h36m59.433190795s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562234630 (496h36m59.433317676s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.562980201 (496h36m59.434063246s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564012245 (496h36m59.43509529s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.564629551 (496h36m59.435712596s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:05:59.565120215 (496h36m59.43620325s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Uploading piece... Dec 26 08:06:01.176850499 (496h37m1.047933554s) [upload=435884]: Next returned nil segment Dec 26 08:06:01.176869015 (496h37m1.04795206s) [upload=435884]: Waiting for error group managing segments... Dec 26 08:06:01.560358332 (496h37m1.431441387s) [noise=true , address=145.79.227.13:4804 , piece_id=DVUSNNKFGHZXXYDEVPBGR5EUDRVNW7AEURPL4ZDVRIH6VNM5AROA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.561209874 (496h37m1.432292919s) [noise=true , address=145.79.227.49:4337 , piece_id=CSVQLZMXHH4QQMRXR3GYQRE4VX32U3MKDMQE2JSBIJO2VNPLAOYQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.567872280 (496h37m1.438955325s) [noise=true , address=145.79.227.50:4375 , piece_id=KHSB7252CEZC2KCH7GCOHGTD4YYIURKEXOFZTG6H2F6KSPBZHKHQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.568944903 (496h37m1.440027948s) [noise=true , address=145.79.227.6:4257 , piece_id=62R6V7SAKK7L5TZMLN575U3QU5WILVAVODXGOTDRJM4IDJEUA44A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.570504894 (496h37m1.441587939s) [noise=true , address=145.79.227.51:4409 , piece_id=6H7QH3HQEKXE6VHAQH3WYTEUMH2XUGQNCKQOLUZB5AGCNUT3OFZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588111790 (496h37m1.459194836s) [noise=true , address=145.79.227.47:4265 , piece_id=QEVE6OWAKOVKPNSE43JKBGPBHFWEJTZZXUQ7CMFMO6T5QOKOKRUQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.588807835 (496h37m1.45989088s) [noise=true , address=145.79.227.46:4603 , piece_id=A36DHULTXZA34JDXQLNBDDRC67WRQLB4AGNP3TR5BRRC4C5QYJXA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.591812196 (496h37m1.462895241s) [noise=true , address=145.79.227.37:4566 , piece_id=PLWSHMUEXBNUVKPZDX2JSIYLFOP2MZTGTREUF24GDAXHMRJ6TY6Q, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.594307557 (496h37m1.465390602s) [noise=true , address=145.79.227.34:4907 , piece_id=AENBJXGEVBPVX3OXN3X2SHAQYDTFPTSRNVU63LTSEH6OHMEX6B3A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.597042546 (496h37m1.468125591s) [noise=true , address=145.79.227.5:4221 , piece_id=NPHNREBGC2EKEJC4MHPHXUUMQDQFXIEUH5RWTIZALFYDIWJQC3YA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.763606471 (496h37m1.634689516s) [noise=true , address=77.90.31.10:4518 , piece_id=A25FAUJJKPLYJSKPIJBJPK7RDBJYA72WRMSNT7FU5G6QANDGK6NQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:01.911658847 (496h37m1.782741892s) [noise=true , address=45.91.116.6:4974 , piece_id=ZT7EZCK3IZQWHJKPNY7STC6B3NRTBEBALY5IFQJSLKVJR2XJZ5UQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032411771 (496h37m1.903494826s) [noise=true , address=145.79.227.17:4971 , piece_id=RULNG7N5UCANXBZQ5M3EG46G47PZVPMSPLQZMC7ST3VGQAFRAKZA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: <nil> Dec 26 08:06:02.032436208 (496h37m1.903519253s) [seg_pos={0 0}, upload=435884]: Segment reached optimal threshold of 13 pieces. Dec 26 08:06:02.032691944 (496h37m1.90377499s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36232->45.91.116.14:4190: use of closed network connection Dec 26 08:06:02.032701914 (496h37m1.903784959s) [noise=true , address=45.91.116.14:4190 , piece_id=5P76IESJZFYQYVTDX5Y5S6QKSU2NDNNBYTGV7YCLBKQN4SIZXB6A, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032730317 (496h37m1.903813373s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 08:06:02.032740047 (496h37m1.903823102s) [noise=true , address=45.91.116.12:4228 , piece_id=ORZ3UTR66RA4XFQT5EG72LL6USUFAV4WHGT5RYNYY6BMPHRL5RPQ, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032853382 (496h37m1.903936427s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:44234->77.90.31.11:4553: use of closed network connection; storage node overloaded, request limit: 2000 Dec 26 08:06:02.032861668 (496h37m1.903944723s) [noise=true , address=77.90.31.11:4553 , piece_id=FUCYDRRIXV45VFHE7CMQ66DOAIBDUN4RQ2XO55WSX3EOYKORIVFA, seg_pos={0 0}, upload=435884]: Long tail cancellation due to optimal threshold Dec 26 08:06:02.032948333 (496h37m1.904031388s) [seg_pos={0 0}, upload=435884]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 08:06:02.033105061 (496h37m1.904188116s) [upload=435884]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 08:06:02.033140599 (496h37m1.904223654s) [upload=435884]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 08:06:02.328202557 (496h37m2.199285602s) [upload=435884]: Done uploading segments... Dec 26 08:06:02.328232714 (496h37m2.199315759s) [upload=435884]: Upload finished. err: <nil> Duration: 30.002340971s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..X..X.....XX....X...X.....X.....XXX.X......................] duration: 5.971069234s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:05:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.971096256s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...X..X.....XX....X...X.....X.....XXX.X.....................] duration: 5.77239613s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:04:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.772430665s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [....X..X.....XX....X...X.....X.....XXX.X....................] duration: 6.122469868s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:03:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.122497451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [.....X..X.....XX....X...X.....X.....XXX.X...................] duration: 7.835332093s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 7.835359604s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [......X..X.....XX....X...X.....X.....XXX.X..................] duration: 7.936156709s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:01:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 7.936211592s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [X......X..X.....XX....X...X.....X.....XXX.X.................] duration: 6.00744439s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:01:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.007474497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.X......X..X.....XX....X...X.....X.....XXX.X................] duration: 5.857359905s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:00:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.857388521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..X......X..X.....XX....X...X.....X.....XXX.X...............] duration: 20.120307383s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 08:00:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 20.120333784s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X......X..X.....XX....X...X.....X.....XXX.X..............] duration: 5.884469195s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:59:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.884523718s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X......X..X.....XX....X...X.....X.....XXX.X.............] duration: 5.960796755s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:59:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.960821273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X......X..X.....XX....X...X.....X.....XXX.X............] duration: 6.022637253s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:59:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.022666148s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [......X......X..X.....XX....X...X.....X.....XXX.X...........] duration: 5.75030329s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:58:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.750334572s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.......X......X..X.....XX....X...X.....X.....XXX.X..........] duration: 5.919980716s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:57:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.920009992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [........X......X..X.....XX....X...X.....X.....XXX.X.........] duration: 5.707494155s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:56:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.70751181s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.........X......X..X.....XX....X...X.....X.....XXX.X........] duration: 5.797254605s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:55:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.797280855s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..........X......X..X.....XX....X...X.....X.....XXX.X.......] duration: 5.985645169s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:54:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.985683732s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...........X......X..X.....XX....X...X.....X.....XXX.X......] duration: 6.555826294s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:54:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.555855589s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [............X......X..X.....XX....X...X.....X.....XXX.X.....] duration: 6.105230551s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:52:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.105256212s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.............X......X..X.....XX....X...X.....X.....XXX.X....] duration: 5.829394654s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:52:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.829425483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............X......X..X.....XX....X...X.....X.....XXX.X...] duration: 6.357963589s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:51:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 6.358004677s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............X......X..X.....XX....X...X.....X.....XXX.X..] duration: 5.798283001s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:50:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.798312067s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................X......X..X.....XX....X...X.....X.....XXX.X.] duration: 5.880767647s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:50:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 5.880795338s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................X......X..X.....XX....X...X.....X.....XXX.X] duration: 30.003040913s trace: https://jaeger.dev.storj.io/trace/57cf471084dd8e12 FAIL (2025-12-26 07:49:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:48:58.939383608 (496h19m58.810466674s) [upload=435638]: Starting upload Dec 26 07:48:58.939453522 (496h19m58.810536567s) [upload=435638]: Done starting upload Dec 26 07:48:58.939954645 (496h19m58.81103768s) [upload=435638]: Uploading segments... Dec 26 07:48:58.940005362 (496h19m58.811088398s) [upload=435638]: Got next segment. Inline: false Dec 26 07:48:58.940033687 (496h19m58.811116722s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:48:58.940038926 (496h19m58.811121982s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:48:59.184546149 (496h19m59.055629185s) [seg_pos={0 0}, upload=435638]: Begin upload segment... Dec 26 07:48:59.184606233 (496h19m59.055689278s) [seg_pos={0 0}, upload=435638]: Done begin upload segment. Dec 26 07:48:59.184643215 (496h19m59.05572626s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.184706955 (496h19m59.05578999s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185194063 (496h19m59.056277098s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185648628 (496h19m59.056731673s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185674347 (496h19m59.056757392s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.185993956 (496h19m59.057076991s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186173668 (496h19m59.057256703s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186287575 (496h19m59.05737061s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186622893 (496h19m59.057705938s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186702285 (496h19m59.05778533s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.186996655 (496h19m59.05807969s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187020110 (496h19m59.058103145s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187273773 (496h19m59.058356818s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187421404 (496h19m59.058504439s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.187798151 (496h19m59.058881197s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:48:59.188180690 (496h19m59.059263725s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Uploading piece... Dec 26 07:49:00.359002083 (496h20m0.230085128s) [noise=true , address=77.90.31.18:4805 , piece_id=AU65DW3LUP6NHCUBXPXOLBWFUBXOBERTFNLEMWP3KNZWRALYJFRQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: storage node overloaded, request limit: 2000 Dec 26 07:49:00.895949372 (496h20m0.767032407s) [upload=435638]: Next returned nil segment Dec 26 07:49:00.895964510 (496h20m0.767047545s) [upload=435638]: Waiting for error group managing segments... Dec 26 07:49:01.181028680 (496h20m1.052111725s) [noise=true , address=145.79.227.16:4909 , piece_id=SCCEZRWVPANIB36XHHLQGX7XZW5I4QDN66EA2J4BRJZIOKRNS6NQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.181844945 (496h20m1.05292798s) [noise=true , address=145.79.227.6:4257 , piece_id=YMCMGOGKV5QTBBQ7HFMBAJW5DZVBIVR6FFROYJMSQGTV7SNTSXDA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.187902550 (496h20m1.058985595s) [noise=true , address=145.79.227.48:4301 , piece_id=WRMOXRSX6BOMWXZLALIQ7YUSKJBD4JNHLEAXFAWTWHMCUQLDGSZQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.196247209 (496h20m1.067330254s) [noise=true , address=145.79.227.38:4978 , piece_id=NJXIVVNX3T4IWSZZM7ISPXWDXPK7GIXXKANR3IP7XA4FLUDSJ22A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.199243975 (496h20m1.07032702s) [noise=true , address=145.79.227.2:4115 , piece_id=HEE7QONNPX6BGCELQ7QIWXCH5M6KFZIEXXSHLKNINXUZQPIR4RJA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.201331400 (496h20m1.072414445s) [noise=true , address=145.79.227.18:4006 , piece_id=4MZZBKYUA4IRPKRIWLSC7EB2VFF4ZFH3R4735IF46Q56YMPONLWA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.203370954 (496h20m1.074453999s) [noise=true , address=145.79.227.42:4123 , piece_id=7ZAT3XMA46S6JEMS3QDRSZUODZYDDXALNHKY7UFHBZLIIZHZXDKA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.207100886 (496h20m1.078183921s) [noise=true , address=145.79.227.24:4583 , piece_id=QLKII2YJLJ6YFL273WF5QBYI7CAX3N6F25MJP4HWTNRESKNXRXXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.208695433 (496h20m1.079778478s) [noise=true , address=145.79.227.7:4295 , piece_id=EV4RP72C53Z55VCFQNQQTBZXBIC3FNW26OL7KJUG3YWXSCR6RM7A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.214002548 (496h20m1.085085603s) [noise=true , address=145.79.227.12:4765 , piece_id=Q3HVENGGDMS36FHYDDB6JQK3BOW6BWQYGFYPBTSZM323RYHYPVXQ, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.400978102 (496h20m1.272061147s) [noise=true , address=77.90.31.21:4880 , piece_id=NTIM3HYJBSFTL3KR7XUBDDELITMZUPMAEK5W7RUS3ETJZOGMYX7Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:01.579467847 (496h20m1.450550892s) [noise=true , address=45.91.116.10:4083 , piece_id=VKSYNBD3EMCXKL7AIZV5NMYHFWB5OEIMNOI4X3H2IBMLB2YJ55HA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326308357 (496h20m2.197391403s) [noise=true , address=45.91.116.14:4190 , piece_id=IYR6CIVP5PDN5F6JTDVTEXGCSYV55QH3GCHZPJHKSAMP4QODTB3Q, seg_pos={0 0}, upload=435638]: Done uploading piece. err: <nil> Dec 26 07:49:02.326334938 (496h20m2.197417983s) [seg_pos={0 0}, upload=435638]: Segment reached optimal threshold of 13 pieces. Dec 26 07:49:02.326859848 (496h20m2.197942903s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326874375 (496h20m2.19795742s) [noise=true , address=145.79.227.40:4050 , piece_id=CMIBKOB4YVGBI22O4NQTUBTMW7BZLCKVCHAXJTXSNBQMOHR5GWYA, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326883874 (496h20m2.197966929s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF Dec 26 07:49:02.326901186 (496h20m2.197984241s) [noise=true , address=145.79.227.34:4907 , piece_id=6CVZISERXO42MFQ5EVRNHUETSK72AWGUVBH46JK2S3YB7542F64A, seg_pos={0 0}, upload=435638]: Long tail cancellation due to optimal threshold Dec 26 07:49:02.326949768 (496h20m2.198032813s) [seg_pos={0 0}, upload=435638]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:49:02.327124962 (496h20m2.198208017s) [upload=435638]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:49:02.327136935 (496h20m2.19821998s) [upload=435638]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:49:02.623820420 (496h20m2.494903475s) [upload=435638]: Done uploading segments... Dec 26 07:49:02.623849546 (496h20m2.494932591s) [upload=435638]: Upload finished. err: <nil> Duration: 30.00305984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................X......X..X.....XX....X...X.....X.....XXX.] duration: 5.973687384s trace: https://jaeger.dev.storj.io/trace/7ab4a71dc31e0d10 FAIL (2025-12-26 07:48:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA) context deadline exceeded ======LOGS====== Dec 26 07:47:29.741677880 (496h18m29.612760925s) [upload=435611]: Starting upload Dec 26 07:47:29.741763834 (496h18m29.612846879s) [upload=435611]: Done starting upload Dec 26 07:47:29.741843566 (496h18m29.612926601s) [upload=435611]: Uploading segments... Dec 26 07:47:29.742064276 (496h18m29.613147321s) [upload=435611]: Got next segment. Inline: false Dec 26 07:47:29.742112588 (496h18m29.613195634s) [upload=435611]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:47:29.742118419 (496h18m29.613201464s) [upload=435611]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:47:29.984240349 (496h18m29.855323404s) [seg_pos={0 0}, upload=435611]: Begin upload segment... Dec 26 07:47:29.984303319 (496h18m29.855386354s) [seg_pos={0 0}, upload=435611]: Done begin upload segment. Dec 26 07:47:29.984343255 (496h18m29.8554263s) [noise=true , address=45.91.116.13:4154 , piece_id=EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.984414651 (496h18m29.855497696s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.984933319 (496h18m29.856016354s) [noise=true , address=45.91.116.10:4082 , piece_id=EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985212029 (496h18m29.856295064s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985561875 (496h18m29.856644921s) [noise=true , address=145.79.227.41:4085 , piece_id=TFN26WPYPTHER4ZMO4RSAARWU2GKHWSH6UUAMWEYLCUFUQFA5AYQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985892966 (496h18m29.856976001s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986172819 (496h18m29.857255854s) [noise=true , address=145.79.227.14:4838 , piece_id=UW46F3BHQIKDERPXWRG6TNALZ5XHFLCDMNCP7HRDB4NXKGMK3F5Q, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986571167 (496h18m29.857654202s) [noise=true , address=45.91.116.4:4939 , piece_id=ODIBCFX4M4KEIZKLR7UJH7FVRLE3DKZRXXBQYQAL3SJ6L5CM3I4Q, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986948787 (496h18m29.858031832s) [noise=true , address=145.79.227.15:4873 , piece_id=E3ZU2J7EBNVLL2ZOOO6BAAXLVST4IP5YINWPM3IAAS7RRTQ64EMA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.988373700 (496h18m29.859456745s) [noise=true , address=145.79.227.52:4445 , piece_id=4TALVQDK2ONV6QBI2OH7D3NXC7QRKXKMUUXI2POOJF5QHB7B3BGQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.989485627 (496h18m29.860568662s) [noise=true , address=145.79.227.16:4912 , piece_id=5LJUZRF7IUX6TR2IOTPYEUOKADOBNOR253FSI7M45UBKSA32X5GA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.990745747 (496h18m29.861828782s) [noise=true , address=145.79.227.49:4337 , piece_id=KMFYPT6KCHAXRV3Q6DELRZ7OPO7JB33GCVFBB52HHXHGIHYPY3SA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.992010325 (496h18m29.86309336s) [noise=true , address=45.91.116.14:4190 , piece_id=73HRVDSOEY6YUOGLD4X5MBYHO5SQALAKOSWTO72YYFQHHEMUD7VQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.993253922 (496h18m29.864336958s) [noise=true , address=145.79.227.13:4802 , piece_id=VL2YOT64ZO3I4F7GQSGIIATFITRK2ESCPSVTX4LCCJA4LAOQ253A, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.994466922 (496h18m29.865549957s) [noise=true , address=145.79.227.38:4978 , piece_id=75V7LDT7PBU7UZT7URFWPNWGU5FIYA6O4MSJ4XX4RN3DNPREDWZA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.995648132 (496h18m29.866731177s) [noise=true , address=77.90.31.19:4844 , piece_id=XP7WSZX4OWQJV2YMVTYC2RMJHSLUFTWM7FJNR7T6MNCTML7PRVDQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:31.704821738 (496h18m31.575904773s) [upload=435611]: Next returned nil segment Dec 26 07:47:31.704835685 (496h18m31.57591872s) [upload=435611]: Waiting for error group managing segments... Dec 26 07:47:31.984121022 (496h18m31.855204067s) [noise=true , address=145.79.227.49:4337 , piece_id=KMFYPT6KCHAXRV3Q6DELRZ7OPO7JB33GCVFBB52HHXHGIHYPY3SA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:31.989890698 (496h18m31.860973753s) [noise=true , address=145.79.227.41:4085 , piece_id=TFN26WPYPTHER4ZMO4RSAARWU2GKHWSH6UUAMWEYLCUFUQFA5AYQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.005605292 (496h18m31.876688327s) [noise=true , address=145.79.227.38:4978 , piece_id=75V7LDT7PBU7UZT7URFWPNWGU5FIYA6O4MSJ4XX4RN3DNPREDWZA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.009031716 (496h18m31.880114751s) [noise=true , address=145.79.227.13:4802 , piece_id=VL2YOT64ZO3I4F7GQSGIIATFITRK2ESCPSVTX4LCCJA4LAOQ253A, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.011291228 (496h18m31.882374263s) [noise=true , address=145.79.227.52:4445 , piece_id=4TALVQDK2ONV6QBI2OH7D3NXC7QRKXKMUUXI2POOJF5QHB7B3BGQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.014926351 (496h18m31.886009386s) [noise=true , address=145.79.227.15:4873 , piece_id=E3ZU2J7EBNVLL2ZOOO6BAAXLVST4IP5YINWPM3IAAS7RRTQ64EMA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.016877596 (496h18m31.887960641s) [noise=true , address=145.79.227.14:4838 , piece_id=UW46F3BHQIKDERPXWRG6TNALZ5XHFLCDMNCP7HRDB4NXKGMK3F5Q, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.027555408 (496h18m31.898638443s) [noise=true , address=145.79.227.16:4912 , piece_id=5LJUZRF7IUX6TR2IOTPYEUOKADOBNOR253FSI7M45UBKSA32X5GA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.236462032 (496h18m32.107545087s) [noise=true , address=45.91.116.13:4154 , piece_id=EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.280417791 (496h18m32.151500826s) [noise=true , address=77.90.31.19:4844 , piece_id=XP7WSZX4OWQJV2YMVTYC2RMJHSLUFTWM7FJNR7T6MNCTML7PRVDQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.290950437 (496h18m32.162033482s) [noise=true , address=45.91.116.10:4082 , piece_id=EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.327707950 (496h18m32.198790985s) [noise=true , address=45.91.116.14:4190 , piece_id=73HRVDSOEY6YUOGLD4X5MBYHO5SQALAKOSWTO72YYFQHHEMUD7VQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.372450487 (496h18m32.243533522s) [noise=true , address=45.91.116.4:4939 , piece_id=ODIBCFX4M4KEIZKLR7UJH7FVRLE3DKZRXXBQYQAL3SJ6L5CM3I4Q, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.372470465 (496h18m32.24355351s) [seg_pos={0 0}, upload=435611]: Segment reached optimal threshold of 13 pieces. Dec 26 07:47:32.372641531 (496h18m32.243724576s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.372647863 (496h18m32.243730898s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.372904321 (496h18m32.243987366s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.372910422 (496h18m32.243993457s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.373025883 (496h18m32.244108928s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.373031574 (496h18m32.244114609s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.373047714 (496h18m32.244130749s) [seg_pos={0 0}, upload=435611]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:47:32.373110123 (496h18m32.244193158s) [upload=435611]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:47:32.373115824 (496h18m32.244198869s) [upload=435611]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:47:32.656286739 (496h18m32.527369784s) [upload=435611]: Done uploading segments... Dec 26 07:47:32.656310935 (496h18m32.52739398s) [upload=435611]: Upload finished. err: <nil> Duration: 5.973717711s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [...................X......X..X.....XX....X...X.....X.....XXX] duration: 30.004471045s trace: https://jaeger.dev.storj.io/trace/7ab4a71dc31e0d10 FAIL (2025-12-26 07:47:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA) context deadline exceeded; piecestore: (Node ID: 1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, Piece ID: EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA) context deadline exceeded ======LOGS====== Dec 26 07:47:29.741677880 (496h18m29.612760925s) [upload=435611]: Starting upload Dec 26 07:47:29.741763834 (496h18m29.612846879s) [upload=435611]: Done starting upload Dec 26 07:47:29.741843566 (496h18m29.612926601s) [upload=435611]: Uploading segments... Dec 26 07:47:29.742064276 (496h18m29.613147321s) [upload=435611]: Got next segment. Inline: false Dec 26 07:47:29.742112588 (496h18m29.613195634s) [upload=435611]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:47:29.742118419 (496h18m29.613201464s) [upload=435611]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:47:29.984240349 (496h18m29.855323404s) [seg_pos={0 0}, upload=435611]: Begin upload segment... Dec 26 07:47:29.984303319 (496h18m29.855386354s) [seg_pos={0 0}, upload=435611]: Done begin upload segment. Dec 26 07:47:29.984343255 (496h18m29.8554263s) [noise=true , address=45.91.116.13:4154 , piece_id=EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.984414651 (496h18m29.855497696s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.984933319 (496h18m29.856016354s) [noise=true , address=45.91.116.10:4082 , piece_id=EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985212029 (496h18m29.856295064s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985561875 (496h18m29.856644921s) [noise=true , address=145.79.227.41:4085 , piece_id=TFN26WPYPTHER4ZMO4RSAARWU2GKHWSH6UUAMWEYLCUFUQFA5AYQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.985892966 (496h18m29.856976001s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986172819 (496h18m29.857255854s) [noise=true , address=145.79.227.14:4838 , piece_id=UW46F3BHQIKDERPXWRG6TNALZ5XHFLCDMNCP7HRDB4NXKGMK3F5Q, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986571167 (496h18m29.857654202s) [noise=true , address=45.91.116.4:4939 , piece_id=ODIBCFX4M4KEIZKLR7UJH7FVRLE3DKZRXXBQYQAL3SJ6L5CM3I4Q, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.986948787 (496h18m29.858031832s) [noise=true , address=145.79.227.15:4873 , piece_id=E3ZU2J7EBNVLL2ZOOO6BAAXLVST4IP5YINWPM3IAAS7RRTQ64EMA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.988373700 (496h18m29.859456745s) [noise=true , address=145.79.227.52:4445 , piece_id=4TALVQDK2ONV6QBI2OH7D3NXC7QRKXKMUUXI2POOJF5QHB7B3BGQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.989485627 (496h18m29.860568662s) [noise=true , address=145.79.227.16:4912 , piece_id=5LJUZRF7IUX6TR2IOTPYEUOKADOBNOR253FSI7M45UBKSA32X5GA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.990745747 (496h18m29.861828782s) [noise=true , address=145.79.227.49:4337 , piece_id=KMFYPT6KCHAXRV3Q6DELRZ7OPO7JB33GCVFBB52HHXHGIHYPY3SA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.992010325 (496h18m29.86309336s) [noise=true , address=45.91.116.14:4190 , piece_id=73HRVDSOEY6YUOGLD4X5MBYHO5SQALAKOSWTO72YYFQHHEMUD7VQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.993253922 (496h18m29.864336958s) [noise=true , address=145.79.227.13:4802 , piece_id=VL2YOT64ZO3I4F7GQSGIIATFITRK2ESCPSVTX4LCCJA4LAOQ253A, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.994466922 (496h18m29.865549957s) [noise=true , address=145.79.227.38:4978 , piece_id=75V7LDT7PBU7UZT7URFWPNWGU5FIYA6O4MSJ4XX4RN3DNPREDWZA, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:29.995648132 (496h18m29.866731177s) [noise=true , address=77.90.31.19:4844 , piece_id=XP7WSZX4OWQJV2YMVTYC2RMJHSLUFTWM7FJNR7T6MNCTML7PRVDQ, seg_pos={0 0}, upload=435611]: Uploading piece... Dec 26 07:47:31.704821738 (496h18m31.575904773s) [upload=435611]: Next returned nil segment Dec 26 07:47:31.704835685 (496h18m31.57591872s) [upload=435611]: Waiting for error group managing segments... Dec 26 07:47:31.984121022 (496h18m31.855204067s) [noise=true , address=145.79.227.49:4337 , piece_id=KMFYPT6KCHAXRV3Q6DELRZ7OPO7JB33GCVFBB52HHXHGIHYPY3SA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:31.989890698 (496h18m31.860973753s) [noise=true , address=145.79.227.41:4085 , piece_id=TFN26WPYPTHER4ZMO4RSAARWU2GKHWSH6UUAMWEYLCUFUQFA5AYQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.005605292 (496h18m31.876688327s) [noise=true , address=145.79.227.38:4978 , piece_id=75V7LDT7PBU7UZT7URFWPNWGU5FIYA6O4MSJ4XX4RN3DNPREDWZA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.009031716 (496h18m31.880114751s) [noise=true , address=145.79.227.13:4802 , piece_id=VL2YOT64ZO3I4F7GQSGIIATFITRK2ESCPSVTX4LCCJA4LAOQ253A, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.011291228 (496h18m31.882374263s) [noise=true , address=145.79.227.52:4445 , piece_id=4TALVQDK2ONV6QBI2OH7D3NXC7QRKXKMUUXI2POOJF5QHB7B3BGQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.014926351 (496h18m31.886009386s) [noise=true , address=145.79.227.15:4873 , piece_id=E3ZU2J7EBNVLL2ZOOO6BAAXLVST4IP5YINWPM3IAAS7RRTQ64EMA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.016877596 (496h18m31.887960641s) [noise=true , address=145.79.227.14:4838 , piece_id=UW46F3BHQIKDERPXWRG6TNALZ5XHFLCDMNCP7HRDB4NXKGMK3F5Q, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.027555408 (496h18m31.898638443s) [noise=true , address=145.79.227.16:4912 , piece_id=5LJUZRF7IUX6TR2IOTPYEUOKADOBNOR253FSI7M45UBKSA32X5GA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.236462032 (496h18m32.107545087s) [noise=true , address=45.91.116.13:4154 , piece_id=EYAYZHWXFUWYTL2VDPRJRLBB23CHWAZEAYM54OMSOJFSHNAM6UTA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.280417791 (496h18m32.151500826s) [noise=true , address=77.90.31.19:4844 , piece_id=XP7WSZX4OWQJV2YMVTYC2RMJHSLUFTWM7FJNR7T6MNCTML7PRVDQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.290950437 (496h18m32.162033482s) [noise=true , address=45.91.116.10:4082 , piece_id=EHAW4ZRL5JU6KAU75DQ6TEM5ILERYP5M73NVKR5XL2YPZQSWM3AA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.327707950 (496h18m32.198790985s) [noise=true , address=45.91.116.14:4190 , piece_id=73HRVDSOEY6YUOGLD4X5MBYHO5SQALAKOSWTO72YYFQHHEMUD7VQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.372450487 (496h18m32.243533522s) [noise=true , address=45.91.116.4:4939 , piece_id=ODIBCFX4M4KEIZKLR7UJH7FVRLE3DKZRXXBQYQAL3SJ6L5CM3I4Q, seg_pos={0 0}, upload=435611]: Done uploading piece. err: <nil> Dec 26 07:47:32.372470465 (496h18m32.24355351s) [seg_pos={0 0}, upload=435611]: Segment reached optimal threshold of 13 pieces. Dec 26 07:47:32.372641531 (496h18m32.243724576s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.372647863 (496h18m32.243730898s) [noise=true , address=145.79.227.44:4194 , piece_id=MYA6RFB5OCBOBZASO4DJ7HKIUXO2FBMXIZB4I5D7CFWB3VQZODTQ, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.372904321 (496h18m32.243987366s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.372910422 (496h18m32.243993457s) [noise=true , address=45.91.116.5:4298 , piece_id=4R2VGE5M7MI44CFRQPZXMNBHQCCQZAKXSWTNKXCFHZSQPH4QXCXA, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.373025883 (496h18m32.244108928s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:32.373031574 (496h18m32.244114609s) [noise=true , address=145.79.227.5:4221 , piece_id=2RSXZEWPFTZ67MOLIVNVEMG3RXDYD3N4DLG2VJTCKYLU562QMXPQ, seg_pos={0 0}, upload=435611]: Long tail cancellation due to optimal threshold Dec 26 07:47:32.373047714 (496h18m32.244130749s) [seg_pos={0 0}, upload=435611]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:47:32.373110123 (496h18m32.244193158s) [upload=435611]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:47:32.373115824 (496h18m32.244198869s) [upload=435611]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:47:32.656286739 (496h18m32.527369784s) [upload=435611]: Done uploading segments... Dec 26 07:47:32.656310935 (496h18m32.52739398s) [upload=435611]: Upload finished. err: <nil> Duration: 30.004480956s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................X......X..X.....XX....X...X.....X.....XX] duration: 30.00337055s trace: https://jaeger.dev.storj.io/trace/2f62607aeab86c71 FAIL (2025-12-26 07:47:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy, Piece ID: UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA) context deadline exceeded; piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA) context deadline exceeded ======LOGS====== Dec 26 07:46:59.500346031 (496h17m59.371429076s) [upload=435608]: Starting upload Dec 26 07:46:59.500413710 (496h17m59.371496755s) [upload=435608]: Done starting upload Dec 26 07:46:59.500526805 (496h17m59.37160985s) [upload=435608]: Uploading segments... Dec 26 07:46:59.500641164 (496h17m59.371724199s) [upload=435608]: Got next segment. Inline: false Dec 26 07:46:59.500683494 (496h17m59.371766539s) [upload=435608]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:46:59.500691048 (496h17m59.371774093s) [upload=435608]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:46:59.742516015 (496h17m59.61359906s) [seg_pos={0 0}, upload=435608]: Begin upload segment... Dec 26 07:46:59.742592681 (496h17m59.613675716s) [seg_pos={0 0}, upload=435608]: Done begin upload segment. Dec 26 07:46:59.742653606 (496h17m59.613736641s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.743844264 (496h17m59.614927299s) [noise=true , address=145.79.227.48:4302 , piece_id=U3C2MYQQ4IGXJKCFUHKBREOSIUKNGAFITJZOILQQF4H2ADNLATCQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.744515642 (496h17m59.615598687s) [noise=true , address=45.91.116.10:4082 , piece_id=JCXOJBU2IUWTF237I72TULKVGW52PO4Y4MW22Z2EDKUQCKU6FOFA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745012057 (496h17m59.616095102s) [noise=true , address=45.91.116.9:4261 , piece_id=JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745461102 (496h17m59.616544147s) [noise=true , address=145.79.227.24:4583 , piece_id=S746H363TP5DS2SV7VOIDFCYQZYQIVT3ELKEPLLXIP7AAIYNV7IA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.745976073 (496h17m59.617059108s) [noise=true , address=45.91.116.6:4973 , piece_id=OA2XNBRVPFE2B6I5OVYDWZOFBGETL25LW42G2U4SQ74MR7LQ2RUQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.747430483 (496h17m59.618513528s) [noise=true , address=145.79.227.52:4445 , piece_id=B5QBWYOOOYPJAMT2YGNJ4GR7PEF36OSAT6O6RHUUAGXA7R4IENYA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.748858121 (496h17m59.619941166s) [noise=true , address=145.79.227.11:4729 , piece_id=LB5PG4VSM3RQPWBXOLG5JEMGZKCQSBRTL6ZR22ZMT7P6O7QHF2JA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.750208341 (496h17m59.621291386s) [noise=true , address=45.91.116.8:4046 , piece_id=BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.751369263 (496h17m59.622452308s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.751698369 (496h17m59.622781414s) [noise=true , address=145.79.227.13:4802 , piece_id=42I4ZA2XR6AK7WOKKPRHBNWXXKFNHZKREZOOF6VYP5YOY7ZDXIDA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.753333823 (496h17m59.624416868s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.754320722 (496h17m59.625403767s) [noise=true , address=145.79.227.4:4185 , piece_id=KNPJLLAWA7UF2WKEOC7X7XYCUP2MW3FA3WHQPRUKWYULA5XT3FJA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.755810920 (496h17m59.626893955s) [noise=true , address=77.90.31.19:4844 , piece_id=772UP7KOQTBD6ZGYTSPXQ3MCVP64X3PCY3CYQOSQLP5RMNCQMEFQ, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.756576548 (496h17m59.627659593s) [noise=true , address=45.91.116.12:4225 , piece_id=UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:46:59.757777805 (496h17m59.62886085s) [noise=true , address=145.79.227.51:4409 , piece_id=FXOLZRNC5QRTYRUIG4R2WBYNIBYXDGNA5H7R4ZFXZG2CRZ4OCY4Q, seg_pos={0 0}, upload=435608]: Uploading piece... Dec 26 07:47:01.470306200 (496h18m1.341389245s) [upload=435608]: Next returned nil segment Dec 26 07:47:01.470331237 (496h18m1.341414282s) [upload=435608]: Waiting for error group managing segments... Dec 26 07:47:01.755785179 (496h18m1.626868234s) [noise=true , address=145.79.227.11:4729 , piece_id=LB5PG4VSM3RQPWBXOLG5JEMGZKCQSBRTL6ZR22ZMT7P6O7QHF2JA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.761053742 (496h18m1.632136787s) [noise=true , address=145.79.227.48:4302 , piece_id=U3C2MYQQ4IGXJKCFUHKBREOSIUKNGAFITJZOILQQF4H2ADNLATCQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.762628881 (496h18m1.633711926s) [noise=true , address=145.79.227.13:4802 , piece_id=42I4ZA2XR6AK7WOKKPRHBNWXXKFNHZKREZOOF6VYP5YOY7ZDXIDA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.763467428 (496h18m1.634550473s) [noise=true , address=145.79.227.24:4583 , piece_id=S746H363TP5DS2SV7VOIDFCYQZYQIVT3ELKEPLLXIP7AAIYNV7IA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.773924270 (496h18m1.645007315s) [noise=true , address=145.79.227.52:4445 , piece_id=B5QBWYOOOYPJAMT2YGNJ4GR7PEF36OSAT6O6RHUUAGXA7R4IENYA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.778328246 (496h18m1.649411291s) [noise=true , address=145.79.227.51:4409 , piece_id=FXOLZRNC5QRTYRUIG4R2WBYNIBYXDGNA5H7R4ZFXZG2CRZ4OCY4Q, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:01.788908192 (496h18m1.659991237s) [noise=true , address=145.79.227.4:4185 , piece_id=KNPJLLAWA7UF2WKEOC7X7XYCUP2MW3FA3WHQPRUKWYULA5XT3FJA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.014099153 (496h18m1.885182208s) [noise=true , address=45.91.116.8:4046 , piece_id=BKBAY745JKSCGGGYHZCW3CJ6P2QKSR77FJEQ3F6DMPROPGLZO6PA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.014136925 (496h18m1.88521997s) [noise=true , address=45.91.116.9:4261 , piece_id=JENCIWUW2SCKOVT5PU7CT3XDTKBIACFB2IDTUJ7MD7UGKF6KZPJQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.019282854 (496h18m1.890365899s) [noise=true , address=45.91.116.10:4082 , piece_id=JCXOJBU2IUWTF237I72TULKVGW52PO4Y4MW22Z2EDKUQCKU6FOFA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.027214217 (496h18m1.898297262s) [noise=true , address=77.90.31.19:4844 , piece_id=772UP7KOQTBD6ZGYTSPXQ3MCVP64X3PCY3CYQOSQLP5RMNCQMEFQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.102714321 (496h18m1.973797366s) [noise=true , address=45.91.116.12:4225 , piece_id=UCJJDR2Q3LROBKIGRKRTJTZV34ISNTHMG7GLIS7UGQF7YJ5ENSEA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.108561465 (496h18m1.97964451s) [noise=true , address=45.91.116.6:4973 , piece_id=OA2XNBRVPFE2B6I5OVYDWZOFBGETL25LW42G2U4SQ74MR7LQ2RUQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: <nil> Dec 26 07:47:02.108584319 (496h18m1.979667364s) [seg_pos={0 0}, upload=435608]: Segment reached optimal threshold of 13 pieces. Dec 26 07:47:02.108842851 (496h18m1.979925896s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.108854744 (496h18m1.979937789s) [noise=true , address=77.90.31.16:4735 , piece_id=2W2SXBFXOJH4WAZSJKBWFE2MHFBXGPC4TJL2ZQWIVXS3FBJQHXHQ, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.108987055 (496h18m1.9800701s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.109003377 (496h18m1.980086412s) [noise=true , address=145.79.227.33:4870 , piece_id=X7X7OIUE2UTRKOIE3B4X3TUR7HTJTDK6LKVX2VCJZKX7H4AFIHNA, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.109208518 (496h18m1.980291563s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF Dec 26 07:47:02.109218015 (496h18m1.98030106s) [noise=true , address=45.91.116.13:4156 , piece_id=ZWQYKQGH5GQZ3FR4E3KGUTE7BQD62H5DRSSZLM6HKSRDIBAKBGRQ, seg_pos={0 0}, upload=435608]: Long tail cancellation due to optimal threshold Dec 26 07:47:02.109238555 (496h18m1.9803216s) [seg_pos={0 0}, upload=435608]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:47:02.109319278 (496h18m1.980402333s) [upload=435608]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:47:02.109328005 (496h18m1.98041106s) [upload=435608]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:47:02.397450669 (496h18m2.268533724s) [upload=435608]: Done uploading segments... Dec 26 07:47:02.397475306 (496h18m2.268558351s) [upload=435608]: Upload finished. err: <nil> Duration: 30.003405807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....................X......X..X.....XX....X...X.....X.....X] duration: 30.011143452s trace: https://jaeger.dev.storj.io/trace/47d9959386134e9a FAIL (2025-12-26 07:46:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 26 07:46:26.392863497 (496h17m26.263946532s) [upload=435597]: Starting upload Dec 26 07:46:26.392924823 (496h17m26.264007858s) [upload=435597]: Done starting upload Dec 26 07:46:26.393025254 (496h17m26.264108299s) [upload=435597]: Uploading segments... Dec 26 07:46:26.393098223 (496h17m26.264181258s) [upload=435597]: Got next segment. Inline: false Dec 26 07:46:26.393131987 (496h17m26.264215072s) [upload=435597]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:46:26.393137427 (496h17m26.264220462s) [upload=435597]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:46:26.644476871 (496h17m26.515559916s) [seg_pos={0 0}, upload=435597]: Begin upload segment... Dec 26 07:46:26.644551112 (496h17m26.515634157s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6XNXSENYRKSRW5VIGVYSWPHIELFN3WALG2HZWSICU4K4LC57KFQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.644581260 (496h17m26.515664295s) [seg_pos={0 0}, upload=435597]: Done begin upload segment. Dec 26 07:46:26.644623500 (496h17m26.515706545s) [noise=true , address=145.79.227.35:4481 , piece_id=HAAUZMN65N5IRRTSPOQV3TLNAY77YDF5THIUFDEK3QJMIUCVWLDQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.645391492 (496h17m26.516474537s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646214729 (496h17m26.517297774s) [noise=true , address=145.79.227.11:4732 , piece_id=EOF4ZKDGFL2TWNSEK3DBA2HH2FQUOQWVDIW4VOUJZFUWJ3VGTJRQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646499091 (496h17m26.517582137s) [noise=true , address=145.79.227.9:4366 , piece_id=C56TO7GYPSTODXWEN4FOGHRQE3PZLP4SRPTJ6AEDDBNQ3LCV3HLA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646828829 (496h17m26.517911874s) [noise=true , address=145.79.227.16:4910 , piece_id=WXOYNOFYDSKY4LLFSY4ZUIY4MHQYBG7DBOK3R423K7G4MZ4KB5IA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.646842184 (496h17m26.517925219s) [noise=true , address=45.91.116.11:4117 , piece_id=Y7GTJIMDLOV3M4CINNFX6AAFLE642EEOLJCTHXLQFEQDZJUTSMBA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647107540 (496h17m26.518190585s) [noise=true , address=145.79.227.12:4765 , piece_id=M3HAVTTAWSA5AFDNR2JC4QY7IZKWIHMHTMAX56UHYYDRIJVEVQ2A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647223731 (496h17m26.518306766s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647398523 (496h17m26.518481568s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.647679349 (496h17m26.518762394s) [noise=true , address=77.90.31.11:4555 , piece_id=JVD4ZUYEGOIOY2HHYAV7SDE7XDELVJBSSQDLQEYLWPKYGOVZTP4Q, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648000931 (496h17m26.519083986s) [noise=true , address=145.79.227.41:4087 , piece_id=GCGFTL6OQZCKUGQVK5TKH2QFECUCALUXU5VTN7OHJBFQB2NFLYIQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648331480 (496h17m26.519414515s) [noise=true , address=145.79.227.36:4518 , piece_id=AD7RL47BNBZKBBFRWVFF5KTY362L7J7T4H4MO7QNZNFUX7ELR5KQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648656530 (496h17m26.519739565s) [noise=true , address=45.91.116.6:4974 , piece_id=PWBT3CRWVCNHOJF4WFU4SRHOVPZLUCLESTCFGFPC7B472LNDZCWA, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.648996025 (496h17m26.52007907s) [noise=true , address=45.91.116.12:4225 , piece_id=5K4IBGSCVSG4CVZZAAHJEZR5I7TBJTSCEFCMVEY47DUI5QYPZFPQ, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:26.649233828 (496h17m26.520316873s) [noise=true , address=145.79.227.18:4006 , piece_id=L26J4SDWK2D5QJFW7C4XYBBZAZABOBJTQFSE2W7USBHMGYKE6J3A, seg_pos={0 0}, upload=435597]: Uploading piece... Dec 26 07:46:28.364984237 (496h17m28.236067272s) [upload=435597]: Next returned nil segment Dec 26 07:46:28.364998245 (496h17m28.23608128s) [upload=435597]: Waiting for error group managing segments... Dec 26 07:46:28.650501751 (496h17m28.521584786s) [noise=true , address=145.79.227.18:4006 , piece_id=L26J4SDWK2D5QJFW7C4XYBBZAZABOBJTQFSE2W7USBHMGYKE6J3A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.652863779 (496h17m28.523946814s) [noise=true , address=145.79.227.36:4518 , piece_id=AD7RL47BNBZKBBFRWVFF5KTY362L7J7T4H4MO7QNZNFUX7ELR5KQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.654667684 (496h17m28.525750729s) [noise=true , address=145.79.227.12:4765 , piece_id=M3HAVTTAWSA5AFDNR2JC4QY7IZKWIHMHTMAX56UHYYDRIJVEVQ2A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.657515417 (496h17m28.528598452s) [noise=true , address=145.79.227.11:4732 , piece_id=EOF4ZKDGFL2TWNSEK3DBA2HH2FQUOQWVDIW4VOUJZFUWJ3VGTJRQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.657546757 (496h17m28.528629802s) [noise=true , address=145.79.227.17:4970 , piece_id=Q6XNXSENYRKSRW5VIGVYSWPHIELFN3WALG2HZWSICU4K4LC57KFQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.672310940 (496h17m28.543393975s) [noise=true , address=145.79.227.16:4910 , piece_id=WXOYNOFYDSKY4LLFSY4ZUIY4MHQYBG7DBOK3R423K7G4MZ4KB5IA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.674878239 (496h17m28.545961294s) [noise=true , address=145.79.227.41:4087 , piece_id=GCGFTL6OQZCKUGQVK5TKH2QFECUCALUXU5VTN7OHJBFQB2NFLYIQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.681015725 (496h17m28.55209876s) [noise=true , address=145.79.227.35:4481 , piece_id=HAAUZMN65N5IRRTSPOQV3TLNAY77YDF5THIUFDEK3QJMIUCVWLDQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.682868172 (496h17m28.553951208s) [noise=true , address=145.79.227.9:4366 , piece_id=C56TO7GYPSTODXWEN4FOGHRQE3PZLP4SRPTJ6AEDDBNQ3LCV3HLA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.898229648 (496h17m28.769312693s) [noise=true , address=45.91.116.6:4974 , piece_id=PWBT3CRWVCNHOJF4WFU4SRHOVPZLUCLESTCFGFPC7B472LNDZCWA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.916470252 (496h17m28.787553307s) [noise=true , address=45.91.116.12:4225 , piece_id=5K4IBGSCVSG4CVZZAAHJEZR5I7TBJTSCEFCMVEY47DUI5QYPZFPQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.917138945 (496h17m28.78822199s) [noise=true , address=77.90.31.11:4555 , piece_id=JVD4ZUYEGOIOY2HHYAV7SDE7XDELVJBSSQDLQEYLWPKYGOVZTP4Q, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.984535378 (496h17m28.855618413s) [noise=true , address=45.91.116.11:4117 , piece_id=Y7GTJIMDLOV3M4CINNFX6AAFLE642EEOLJCTHXLQFEQDZJUTSMBA, seg_pos={0 0}, upload=435597]: Done uploading piece. err: <nil> Dec 26 07:46:28.984561047 (496h17m28.855644092s) [seg_pos={0 0}, upload=435597]: Segment reached optimal threshold of 13 pieces. Dec 26 07:46:28.984790004 (496h17m28.855873039s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 26 07:46:28.984798900 (496h17m28.855881935s) [noise=true , address=45.91.116.10:4083 , piece_id=RQD4FMCYDKJKWP5CGW5LDWD3JCAQ6EDSDY4YZ6GX4MGADC5375FQ, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985133598 (496h17m28.856216683s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39786->45.91.116.5:4298: use of closed network connection Dec 26 07:46:28.985145079 (496h17m28.856228115s) [noise=true , address=45.91.116.5:4298 , piece_id=QZHMSVZUO6YCLWCIHFNIZOM67FDGJLCFPZINKMOHMKO5V5E3CN3A, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985308421 (496h17m28.856391466s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 26 07:46:28.985316586 (496h17m28.856399621s) [noise=true , address=45.91.116.7:4010 , piece_id=OUJK4NY7OKPW3PTVDMBYRV6XONDVDDNEMPTU3UW5RT4NNTWLC66Q, seg_pos={0 0}, upload=435597]: Long tail cancellation due to optimal threshold Dec 26 07:46:28.985338397 (496h17m28.856421432s) [seg_pos={0 0}, upload=435597]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:46:28.985423600 (496h17m28.856506645s) [upload=435597]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:46:28.985430182 (496h17m28.856513237s) [upload=435597]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:46:29.294738116 (496h17m29.165821162s) [upload=435597]: Done uploading segments... Dec 26 07:46:29.294757394 (496h17m29.165840429s) [upload=435597]: Upload finished. err: <nil> Duration: 30.011156998s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [......................X......X..X.....XX....X...X.....X.....] duration: 5.766312255s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:46:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.766346439s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.......................X......X..X.....XX....X...X.....X....] duration: 5.729133821s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:45:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.729152126s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [........................X......X..X.....XX....X...X.....X...] duration: 5.759622005s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:44:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.759651311s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [.........................X......X..X.....XX....X...X.....X..] duration: 5.73062238s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:43:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.730648068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [..........................X......X..X.....XX....X...X.....X.] duration: 5.956819977s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:42:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 5.956852891s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........................X......X..X.....XX....X...X.....X] duration: 30.004144679s trace: https://jaeger.dev.storj.io/trace/3ada0bce4c804493 FAIL (2025-12-26 07:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ) context deadline exceeded ======LOGS====== Dec 26 07:41:20.772857051 (496h12m20.643940096s) [upload=435526]: Starting upload Dec 26 07:41:20.773002639 (496h12m20.644085684s) [upload=435526]: Done starting upload Dec 26 07:41:20.773086909 (496h12m20.644169954s) [upload=435526]: Uploading segments... Dec 26 07:41:20.773226605 (496h12m20.64430965s) [upload=435526]: Got next segment. Inline: false Dec 26 07:41:20.773264928 (496h12m20.644347973s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:41:20.773270759 (496h12m20.644353804s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:41:21.015632878 (496h12m20.886715923s) [seg_pos={0 0}, upload=435526]: Begin upload segment... Dec 26 07:41:21.015686049 (496h12m20.886769084s) [seg_pos={0 0}, upload=435526]: Done begin upload segment. Dec 26 07:41:21.015716417 (496h12m20.886799452s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018118020 (496h12m20.889201075s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018197081 (496h12m20.889280116s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018894800 (496h12m20.889977835s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.018914687 (496h12m20.889997732s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019243664 (496h12m20.890326699s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019282969 (496h12m20.890366014s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019590975 (496h12m20.89067402s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019652301 (496h12m20.890735336s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.019997529 (496h12m20.891080574s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020077842 (496h12m20.891160877s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020312478 (496h12m20.891395524s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020503312 (496h12m20.891586347s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020705257 (496h12m20.891788292s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.020843320 (496h12m20.891926365s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:21.021046827 (496h12m20.892129872s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Uploading piece... Dec 26 07:41:22.726750273 (496h12m22.597833308s) [upload=435526]: Next returned nil segment Dec 26 07:41:22.726764761 (496h12m22.597847806s) [upload=435526]: Waiting for error group managing segments... Dec 26 07:41:23.010722415 (496h12m22.88180545s) [noise=true , address=145.79.227.9:4366 , piece_id=K5ANJJZ53WWFUQ6ZABFRGCUHYWCPENZ2GD26AFGHOJW6Z62PZO7A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.026301129 (496h12m22.897384174s) [noise=true , address=145.79.227.37:4565 , piece_id=ISJJRJ6GLDBOBPI3ERIBCXA424EIR3VOXXXEXFK76DVN5BSK62FA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.027049544 (496h12m22.898132589s) [noise=true , address=145.79.227.17:4970 , piece_id=KKOSPNRYJYUY7BWBJIIAZ34BR66MNBANKOLK2RFCHDPR2QEH7JYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.034344515 (496h12m22.90542756s) [noise=true , address=145.79.227.35:4482 , piece_id=3QT3VB752DZYHUS6HTVTH76KKK6Y6DVLYLKIMSMHXQ56QROHPJVA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.037773374 (496h12m22.908856409s) [noise=true , address=145.79.227.19:4077 , piece_id=NHY76X3AOTK5XGKV2N3DKGKAH4PVM3IWDSMXCPAULEU5NTHRBB3A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040538489 (496h12m22.911621524s) [noise=true , address=145.79.227.41:4087 , piece_id=TFEWTNXE3EZLA7EHKATVN73ZO2FAKD7TBIR26YISUQSD53IQVSTQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.040999406 (496h12m22.912082441s) [noise=true , address=145.79.227.15:4875 , piece_id=LCT7S5MWYJPJHJTLBKNWLDGEVEMDAP4N6LZT3P6Y3QHN53SEZG2A, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.053214276 (496h12m22.924297321s) [noise=true , address=145.79.227.11:4729 , piece_id=4ITFMWLGZHEPTOS5J2PMGF4N6VUGH2HSBUD3I7JNYAIW3VNBX5WA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.244215003 (496h12m23.115298038s) [noise=true , address=45.91.116.10:4084 , piece_id=UN4ECTJZ4LBXGX7FKQWGZUSMLRD2CA6BK2ZKDP7ZQJJUZR7O227Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.269369482 (496h12m23.140452527s) [noise=true , address=45.91.116.11:4120 , piece_id=PXKBVGHB4DMH3PH32AWRUKSTIZQ4OKXQXDXCS4NA64J46IC3C5QQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.342663666 (496h12m23.213746701s) [noise=true , address=77.90.31.13:4627 , piece_id=RTC7PGUK7OYO7CQMEMHRKVCL2RAKMDVD3ETNDFLNYOKMW75EDMDA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.369958430 (496h12m23.241041465s) [noise=true , address=45.91.116.6:4974 , piece_id=MDUCJCNGLO4HHJNYK7WE6P4ZP6ZANJ562PIBGLFXN5TP3MLVBE7Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386603475 (496h12m23.25768652s) [noise=true , address=77.90.31.10:4517 , piece_id=3VWDGQ7OZ427MHEM4H4LQWSQA4A3RSJBGCJOS6KVGMPKSEL66ONA, seg_pos={0 0}, upload=435526]: Done uploading piece. err: <nil> Dec 26 07:41:23.386624975 (496h12m23.257708011s) [seg_pos={0 0}, upload=435526]: Segment reached optimal threshold of 13 pieces. Dec 26 07:41:23.386900370 (496h12m23.257983405s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.386906732 (496h12m23.257989777s) [noise=true , address=45.91.116.8:4045 , piece_id=OPP6QGSGTCEEZM2KYCEC7KJU75K7ZFDPLXVCYXEAYURNOXTXGW3Q, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387125319 (496h12m23.258208364s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387131611 (496h12m23.258214656s) [noise=true , address=45.91.116.9:4264 , piece_id=TEEY54JL2JMPWZ775UCCOOZFADKTHM3AXFUWTIPCJLEZENXM7EYQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387235579 (496h12m23.258318624s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 26 07:41:23.387240678 (496h12m23.258323713s) [noise=true , address=45.91.116.12:4228 , piece_id=QFCKU4R6ONPA2RJVMGRILAHOYWSQEANSGB4SPLSBAZKRAHYHLCJQ, seg_pos={0 0}, upload=435526]: Long tail cancellation due to optimal threshold Dec 26 07:41:23.387254354 (496h12m23.258337399s) [seg_pos={0 0}, upload=435526]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:41:23.387325309 (496h12m23.258408354s) [upload=435526]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:41:23.387329718 (496h12m23.258412753s) [upload=435526]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:41:23.672257488 (496h12m23.543340533s) [upload=435526]: Done uploading segments... Dec 26 07:41:23.672276555 (496h12m23.54335959s) [upload=435526]: Upload finished. err: <nil> Duration: 30.004167382s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.414622435s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:41:24 UTC) orbiter warning: context deadline exceeded Duration: 3.414651813s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [............................X......X..X.....XX....X...X.....] duration: 5.730870463s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:41:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 5.730905089s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.514861388s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:40:29 UTC) orbiter warning: context deadline exceeded Duration: 3.514930709s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.459891956s trace: https://jaeger.dev.storj.io/trace/c62fa8a28934b27 FAIL (2025-12-26 07:39:56 UTC) orbiter warning: context deadline exceeded Duration: 3.459926451s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 7 out of 60 is failed (threshold=3) [.............................X......X..X.....XX....X...X....] duration: 5.749704635s trace: https://jaeger.dev.storj.io/trace/49606097c63027bd FAIL (2025-12-26 07:39:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 26 07:36:49.810509099 (496h7m49.681592134s) [upload=435465]: Starting upload Dec 26 07:36:49.810581959 (496h7m49.681664994s) [upload=435465]: Done starting upload Dec 26 07:36:49.810629178 (496h7m49.681712223s) [upload=435465]: Uploading segments... Dec 26 07:36:49.810813389 (496h7m49.681896424s) [upload=435465]: Got next segment. Inline: false Dec 26 07:36:49.810865168 (496h7m49.681948203s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 26 07:36:49.810871069 (496h7m49.681954104s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 26 07:36:50.052860006 (496h7m49.923943061s) [seg_pos={0 0}, upload=435465]: Begin upload segment... Dec 26 07:36:50.052929278 (496h7m49.924012323s) [seg_pos={0 0}, upload=435465]: Done begin upload segment. Dec 26 07:36:50.052982609 (496h7m49.924065644s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.052998590 (496h7m49.924081625s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053208039 (496h7m49.924291084s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.053691289 (496h7m49.924774334s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054083396 (496h7m49.925166441s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.054764151 (496h7m49.925847196s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055141320 (496h7m49.926224355s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055491526 (496h7m49.926574561s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.055937826 (496h7m49.927020872s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.056503834 (496h7m49.927586879s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.058616026 (496h7m49.929699061s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.061265451 (496h7m49.932348486s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.063396549 (496h7m49.934479594s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.068697834 (496h7m49.939780869s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.071698828 (496h7m49.942781863s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:50.073358719 (496h7m49.944441754s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Uploading piece... Dec 26 07:36:51.777718056 (496h7m51.648801091s) [upload=435465]: Next returned nil segment Dec 26 07:36:51.777732173 (496h7m51.648815208s) [upload=435465]: Waiting for error group managing segments... Dec 26 07:36:52.067965365 (496h7m51.9390484s) [noise=true , address=145.79.227.11:4732 , piece_id=DKUSZAJ4RVJCJX253AKEDUS4Z6PPFKIEOZY54KYT7IKZ7H6OTOLA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068318799 (496h7m51.939401844s) [noise=true , address=145.79.227.4:4186 , piece_id=EXUB6Q5LUMXHNTGT5HF7UHI2LXESAVF757GZZFGYWPLUE5KUKLHA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.068694063 (496h7m51.939777108s) [noise=true , address=145.79.227.37:4565 , piece_id=L735BCN2AXMENM2J77MHG4P3AD54MX6BTYTJ74JVGMMOJYS4QIGQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.078694967 (496h7m51.949778002s) [noise=true , address=145.79.227.48:4301 , piece_id=7VQKDXL6CSSBYB45I7QVJIVVPR5TUAOWR4LLDXX3V7OPR7QE7DTA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.083399695 (496h7m51.954482751s) [noise=true , address=145.79.227.42:4121 , piece_id=AHKW22SOXXBOMIRBS3RD73ADZCEIFAFEHQKZVP2H4IMRREIOLKCA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.084777699 (496h7m51.955860734s) [noise=true , address=145.79.227.52:4445 , piece_id=26XLBE4KWIENJGWGTFU7OTM5FJEKUPZSA4QVLZA65FQMCDLWXV2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.094937645 (496h7m51.96602068s) [noise=true , address=145.79.227.17:4972 , piece_id=DCXGPS6TZUCFCRPPLCZVPRCEXQHHBJSUNYFFRD7RLKGBUTRP2L5A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.100793025 (496h7m51.97187606s) [noise=true , address=145.79.227.18:4006 , piece_id=XMV2VG4QTYSP2SKA23HDE54JOEAXJYCPJMME3MRA6GUMA3FCBCHQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.305127051 (496h7m52.176210086s) [noise=true , address=45.91.116.8:4048 , piece_id=4FZAQK3SEXRIIZMITM4WL6OGJHGCCJE2N3FX6Q4FHH73TPQGJRQQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.340648190 (496h7m52.211731235s) [noise=true , address=77.90.31.16:4736 , piece_id=P467UKAF5FN3DB4K5X7OB2LGN56M5PUYI66PN6ZLOPN743WWHIPQ, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.399989786 (496h7m52.271072831s) [noise=true , address=45.91.116.7:4009 , piece_id=KOCNFSIGTHQQLESDH3J43BVNDNP75CAELAWVM64SLTCME2K3FWMA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.408673240 (496h7m52.279756316s) [noise=true , address=45.91.116.13:4153 , piece_id=GVCFMMUGWO56A2Y57BOSDG4ZS2VYKSR3JWQQLK65OSHZ6W674A2Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410008123 (496h7m52.281091158s) [noise=true , address=45.91.116.9:4261 , piece_id=XY37QOSSVHOSJX6F4SOUBWLICSBCEMGI2XOFHQSY2TDO7HVNAA3A, seg_pos={0 0}, upload=435465]: Done uploading piece. err: <nil> Dec 26 07:36:52.410019124 (496h7m52.281102159s) [seg_pos={0 0}, upload=435465]: Segment reached optimal threshold of 13 pieces. Dec 26 07:36:52.410261245 (496h7m52.28134428s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410270131 (496h7m52.281353166s) [noise=true , address=77.90.31.21:4878 , piece_id=OFI5AU36R7LKD44UWL34UPHUV4EB2H4F2KOSTUYHM6JQMFMXZ63Q, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410630658 (496h7m52.281713693s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46838->45.91.116.5:4298: use of closed network connection Dec 26 07:36:52.410638744 (496h7m52.281721779s) [noise=true , address=45.91.116.5:4298 , piece_id=H3THVNLITVZUNGJDFZNIAF4WIIWZQ63GH7T34LMD37Q2ZUVHNRGA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410760285 (496h7m52.28184333s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 26 07:36:52.410765926 (496h7m52.281848961s) [noise=true , address=45.91.116.10:4084 , piece_id=2GAI73QG5C7NR6WRMDCOESHE7QVL4LCXLHJ5OBH4IXLZXQJBACUA, seg_pos={0 0}, upload=435465]: Long tail cancellation due to optimal threshold Dec 26 07:36:52.410781555 (496h7m52.2818646s) [seg_pos={0 0}, upload=435465]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 26 07:36:52.410848994 (496h7m52.281932029s) [upload=435465]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 26 07:36:52.410853142 (496h7m52.281936187s) [upload=435465]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 26 07:36:52.703019295 (496h7m52.574102341s) [upload=435465]: Done uploading segments... Dec 26 07:36:52.703038171 (496h7m52.574121216s) [upload=435465]: Upload finished. err: <nil> Duration: 5.749733911s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)